[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:15 dist-test-slave-19nl krb5kdc[5734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:15 dist-test-slave-19nl krb5kdc[5734](info): set up 2 sockets
Sep 27 18:34:15 dist-test-slave-19nl krb5kdc[5734](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 18:34:16.086205 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:16.195734 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36245
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:16.273666 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:16.273859 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:16.273880 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:16.275044 5747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:16.275084 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:16.275098 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:16.275147 5747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:16.275172 5747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:16.276608 5747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:16.276851 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:16.277084 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:16.281908 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:16.281900 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:16.281963 5747 server_base.cc:1034] running on GCE node
W20260927 18:34:16.281900 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:16.282367 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:16.282661 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:16.283793 5747 hybrid_clock.cc:648] HybridClock initialized: now 1790534056283780 us; error 30 us; skew 500 ppm
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:16.286114 5747 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:16.286613 5747 webserver.cc:460] Webserver started at http://127.5.151.254:35551/ using document root <none> and password file <none>
I20260927 18:34:16.286862 5747 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:16.286907 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:16.287079 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:16.288079 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "a86cba09259647b39213b647331c08ca"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "bce48e955fbe0cbb56c8b46698c4d255"
server_key_iv: "71736bcf4a64b3d9d19078faeff78bc5"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.288419 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "a86cba09259647b39213b647331c08ca"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "bce48e955fbe0cbb56c8b46698c4d255"
server_key_iv: "71736bcf4a64b3d9d19078faeff78bc5"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.289712 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:16.290393 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.290585 5747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:16.290668 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "a86cba09259647b39213b647331c08ca"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "bce48e955fbe0cbb56c8b46698c4d255"
server_key_iv: "71736bcf4a64b3d9d19078faeff78bc5"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.290720 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:16.332175 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:16.334401 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:16.334508 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:16.338272 5747 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:36245
I20260927 18:34:16.338317 5814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:36245 every 8 connection(s)
I20260927 18:34:16.338678 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:16.339599 5815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:16.340397 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 5747
I20260927 18:34:16.340490 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:16.341071 5727 external_mini_cluster.cc:1428] Setting key 96cea4bf759426917ce29e4cb2eef87f
I20260927 18:34:16.342109 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca: Bootstrap starting.
I20260927 18:34:16.342715 5815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:16.342973 5815 log.cc:824] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:16.343606 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca: No bootstrap required, opened a new log
I20260927 18:34:16.345206 5815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86cba09259647b39213b647331c08ca" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36245 } }
I20260927 18:34:16.345326 5815 raft_consensus.cc:374] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:16.345347 5815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86cba09259647b39213b647331c08ca, State: Initialized, Role: FOLLOWER
I20260927 18:34:16.345486 5815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86cba09259647b39213b647331c08ca" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36245 } }
I20260927 18:34:16.345543 5815 raft_consensus.cc:388] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:16.345587 5815 raft_consensus.cc:482] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:16.345633 5815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:16.346446 5815 raft_consensus.cc:504] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86cba09259647b39213b647331c08ca" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36245 } }
I20260927 18:34:16.346596 5815 leader_election.cc:302] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a86cba09259647b39213b647331c08ca; no voters:
I20260927 18:34:16.346766 5815 leader_election.cc:288] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:16.346797 5820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:16.346913 5820 raft_consensus.cc:686] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [term 1 LEADER]: Becoming Leader. State: Replica: a86cba09259647b39213b647331c08ca, State: Running, Role: LEADER
I20260927 18:34:16.347013 5815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:16.347018 5820 consensus_queue.cc:237] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86cba09259647b39213b647331c08ca" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36245 } }
I20260927 18:34:16.347323 5820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a86cba09259647b39213b647331c08ca. Latest consensus state: current_term: 1 leader_uuid: "a86cba09259647b39213b647331c08ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86cba09259647b39213b647331c08ca" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36245 } } }
I20260927 18:34:16.347385 5820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:16.347570 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a86cba09259647b39213b647331c08ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86cba09259647b39213b647331c08ca" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36245 } } }
I20260927 18:34:16.347602 5824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:16.347640 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:16.348228 5824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:16.350639 5824 catalog_manager.cc:1347] Generated new cluster ID: 662253fe18904bb9a7b397754dbe74d5
I20260927 18:34:16.350718 5824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:16.355768 5824 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:16.356364 5824 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:16.359198 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.341901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.5.151.254:36245)
0927 18:34:16.342071 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:16.342080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:16.342339 (+ 259us) server_negotiation.cc:407] Connection header received
0927 18:34:16.344609 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:16.344615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:16.344654 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:16.344780 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:16.346362 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.349110 (+ 2748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.350634 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.351199 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.351297 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.355104 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:16.355109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:16.355109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:16.355122 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:16.357048 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.357469 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.357472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.357472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.357570 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.357789 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.357790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.357790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.358018 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:16.358141 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:16.358831 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 18:34:16.358971 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 18:34:16.366711 5824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a86cba09259647b39213b647331c08ca: Generated new TSK 0
I20260927 18:34:16.366863 5824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:16.393610 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36245
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:16.496457 5841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:16.496656 5841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:16.496678 5841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:16.497805 5841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:16.497869 5841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:16.497913 5841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:16.499488 5841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:16.499711 5841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:16.499935 5841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:16.505590 5849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:16.505566 5847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:16.505825 5841 server_base.cc:1034] running on GCE node
W20260927 18:34:16.505615 5846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:16.506110 5841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:16.506349 5841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:16.507527 5841 hybrid_clock.cc:648] HybridClock initialized: now 1790534056507509 us; error 43 us; skew 500 ppm
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:16.509902 5841 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:16.510318 5841 webserver.cc:460] Webserver started at http://127.5.151.193:36621/ using document root <none> and password file <none>
I20260927 18:34:16.510535 5841 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:16.510637 5841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:16.510787 5841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:16.511863 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "d6dfe61a41a348269a248c493e22a16b"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "ceeb6e651510426a03707d036c9bf39b"
server_key_iv: "3eca36166f1f3c8525fd860e81c84618"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.512213 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "d6dfe61a41a348269a248c493e22a16b"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "ceeb6e651510426a03707d036c9bf39b"
server_key_iv: "3eca36166f1f3c8525fd860e81c84618"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.513549 5841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:16.514263 5856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.514432 5841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:16.514534 5841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "d6dfe61a41a348269a248c493e22a16b"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "ceeb6e651510426a03707d036c9bf39b"
server_key_iv: "3eca36166f1f3c8525fd860e81c84618"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.514634 5841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:16.525928 5841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:16.527886 5841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:16.528045 5841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:16.528262 5841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:16.528551 5841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:16.528600 5841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.528640 5841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:16.528667 5841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.534368 5841 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:40053
I20260927 18:34:16.534435 5969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:40053 every 8 connection(s)
I20260927 18:34:16.534770 5841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:16.538472 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 5841
I20260927 18:34:16.538589 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:16.538757 5727 external_mini_cluster.cc:1428] Setting key e4c1444f3f3a6840295a572946b1d9b1
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:16.544035 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.535449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:39611 (local address 127.5.151.254:36245)
0927 18:34:16.535509 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:16.535511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:16.535705 (+ 194us) server_negotiation.cc:407] Connection header received
0927 18:34:16.536163 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:16.536164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:16.536174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:16.536231 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:16.537144 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.537635 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.538747 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.539032 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.539148 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.541291 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:16.541292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:16.541293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:16.541300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:16.542747 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.543133 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.543134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.543134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.543191 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.543421 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.543422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.543422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.543542 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:16.543612 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:16.543862 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 18:34:16.543938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:34:16.544230 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.535434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36245 (local address 127.5.151.193:39611)
0927 18:34:16.535586 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:16.535625 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:16.536085 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:16.536280 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:16.536283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:16.536821 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:16.537019 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:16.537024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.537713 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.537713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:16.538662 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:16.538666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.539157 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.539159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:16.539290 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.539871 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:16.539879 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:16.541192 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:16.542830 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:16.542837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:16.542839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:16.543052 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:16.543250 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:16.543251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:16.543252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:16.543359 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:16.543602 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:16.543609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:16.543742 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 18:34:16.543841 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:34:16.543988 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260927 18:34:16.544692 5970 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36245
I20260927 18:34:16.544804 5970 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:16.544960 5970 heartbeater.cc:511] Master 127.5.151.254:36245 requested a full tablet report, sending...
I20260927 18:34:16.545513 5779 ts_manager.cc:194] Registered new tserver with Master: d6dfe61a41a348269a248c493e22a16b (127.5.151.193:40053)
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
I20260927 18:34:16.546494 5779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:39611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:16.561762 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36245
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:16.636659 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:16.636865 5975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:16.636919 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:16.637989 5975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:16.638052 5975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:16.638105 5975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:16.639483 5975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:16.639732 5975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:16.639938 5975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:16.644315 5980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:16.644315 5983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:16.644378 5981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:16.644606 5975 server_base.cc:1034] running on GCE node
I20260927 18:34:16.644830 5975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:16.645071 5975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:16.646252 5975 hybrid_clock.cc:648] HybridClock initialized: now 1790534056646235 us; error 30 us; skew 500 ppm
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:16.648926 5975 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:16.649343 5975 webserver.cc:460] Webserver started at http://127.5.151.194:41007/ using document root <none> and password file <none>
I20260927 18:34:16.649529 5975 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:16.649569 5975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:16.649654 5975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:16.650651 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "0dd25d397ea840399ea3c31732f30a2e"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "759ce5f11bb65e1e055ec7d91f48fb86"
server_key_iv: "0bbaf66940b092b68e5b8b5361fdef46"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.650954 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "0dd25d397ea840399ea3c31732f30a2e"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "759ce5f11bb65e1e055ec7d91f48fb86"
server_key_iv: "0bbaf66940b092b68e5b8b5361fdef46"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.652133 5975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:16.652748 5990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.652889 5975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:16.652966 5975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "0dd25d397ea840399ea3c31732f30a2e"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "759ce5f11bb65e1e055ec7d91f48fb86"
server_key_iv: "0bbaf66940b092b68e5b8b5361fdef46"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.653028 5975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:16.664512 5975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:16.666190 5975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:16.666290 5975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:16.666441 5975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:16.666741 5975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:16.666777 5975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.666797 5975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:16.666817 5975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.672266 5975 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:38437
I20260927 18:34:16.672333 6103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:38437 every 8 connection(s)
I20260927 18:34:16.672616 5975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:16.675664 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 5975
I20260927 18:34:16.675766 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:16.675936 5727 external_mini_cluster.cc:1428] Setting key 5fb6cfdb319c74342f74edf33562d1ac
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:16.681576 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.673246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:44311 (local address 127.5.151.254:36245)
0927 18:34:16.673317 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:16.673319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:16.673504 (+ 185us) server_negotiation.cc:407] Connection header received
0927 18:34:16.673963 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:16.673964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:16.673974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:16.674014 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:16.674917 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.675386 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.676482 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.676715 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.676786 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.678911 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:16.678913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:16.678913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:16.678920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:16.680313 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.680699 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.680700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.680701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.680756 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.680981 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.680982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.680982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.681089 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:16.681163 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:16.681407 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 18:34:16.681478 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:34:16.681757 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.673212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36245 (local address 127.5.151.194:44311)
0927 18:34:16.673373 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:16.673406 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:16.673877 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:16.674070 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:16.674073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:16.674671 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:16.674847 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:16.674852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.675452 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.675453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:16.676386 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:16.676389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.676810 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.676811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:16.676935 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.677416 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:16.677424 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:16.678807 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:16.680406 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:16.680413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:16.680415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:16.680623 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:16.680815 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:16.680816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:16.680816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:16.680920 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:16.681155 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:16.681161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:16.681290 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 18:34:16.681387 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:34:16.681532 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260927 18:34:16.682165 6104 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36245
I20260927 18:34:16.682263 6104 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:16.682435 6104 heartbeater.cc:511] Master 127.5.151.254:36245 requested a full tablet report, sending...
I20260927 18:34:16.682870 5779 ts_manager.cc:194] Registered new tserver with Master: 0dd25d397ea840399ea3c31732f30a2e (127.5.151.194:38437)
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
I20260927 18:34:16.683377 5779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:44311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:16.698506 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36245
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:16.775892 6109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:16.776074 6109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:16.776096 6109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:16.777179 6109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:16.777218 6109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:16.777266 6109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:16.778650 6109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:16.778859 6109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:16.779070 6109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:16.783864 6109 server_base.cc:1034] running on GCE node
W20260927 18:34:16.783844 6114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:16.783831 6115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:16.783809 6117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:16.784271 6109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:16.784523 6109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:16.785666 6109 hybrid_clock.cc:648] HybridClock initialized: now 1790534056785654 us; error 28 us; skew 500 ppm
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:16.788057 6109 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:16.788477 6109 webserver.cc:460] Webserver started at http://127.5.151.195:38349/ using document root <none> and password file <none>
I20260927 18:34:16.788702 6109 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:16.788762 6109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:16.788902 6109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:16.789947 6109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "0d009a717a7143c4951056a36601da1d"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "518591ec013789af08bdad39001d5c50"
server_key_iv: "b07f8e4a5c64c46b5c6a4f491d797b6b"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.790315 6109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "0d009a717a7143c4951056a36601da1d"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "518591ec013789af08bdad39001d5c50"
server_key_iv: "b07f8e4a5c64c46b5c6a4f491d797b6b"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.791625 6109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:16.792296 6124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.792435 6109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:16.792536 6109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "0d009a717a7143c4951056a36601da1d"
format_stamp: "Formatted at 2026-09-27 18:34:16 on dist-test-slave-19nl"
server_key: "518591ec013789af08bdad39001d5c50"
server_key_iv: "b07f8e4a5c64c46b5c6a4f491d797b6b"
server_key_version: "encryptionkey@0"
I20260927 18:34:16.792629 6109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:16.812878 6109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:16.814749 6109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:16.814898 6109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:16.815119 6109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:16.815424 6109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:16.815476 6109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.815516 6109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:16.815541 6109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:16.821257 6109 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:34293
I20260927 18:34:16.821303 6237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:34293 every 8 connection(s)
I20260927 18:34:16.821601 6109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:16.822829 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 6109
I20260927 18:34:16.822903 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:16.823092 5727 external_mini_cluster.cc:1428] Setting key 7bafbbc62b1da385229787132a37767a
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:16.830361 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.822338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:50277 (local address 127.5.151.254:36245)
0927 18:34:16.822399 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:16.822401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:16.822600 (+ 199us) server_negotiation.cc:407] Connection header received
0927 18:34:16.823223 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:16.823225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:16.823238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:16.823288 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:16.824194 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.824681 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.825654 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.825858 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.825938 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.827862 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:16.827863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:16.827864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:16.827870 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:16.829079 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.829466 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.829467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.829467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.829519 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.829746 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.829747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.829747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.829886 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:16.829948 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:16.830202 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 18:34:16.830274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:34:16.830547 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.822321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36245 (local address 127.5.151.195:50277)
0927 18:34:16.822471 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:16.822507 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:16.823105 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:16.823361 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:16.823364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:16.823904 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:16.824103 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:16.824107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.824769 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.824770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:16.825566 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:16.825568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.825947 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.825948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:16.826065 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.826476 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:16.826484 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:16.827762 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:16.829172 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:16.829179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:16.829181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:16.829379 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:16.829578 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:16.829579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:16.829580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:16.829678 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:16.829951 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:16.829957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:16.830080 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 18:34:16.830181 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:34:16.830326 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260927 18:34:16.830982 6238 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36245
I20260927 18:34:16.831070 6238 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:16.831235 6238 heartbeater.cc:511] Master 127.5.151.254:36245 requested a full tablet report, sending...
I20260927 18:34:16.831555 5779 ts_manager.cc:194] Registered new tserver with Master: 0d009a717a7143c4951056a36601da1d (127.5.151.195:34293)
I20260927 18:34:16.831913 5779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:50277
I20260927 18:34:16.834818 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:16.855885 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.849026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.5.151.193:40053)
0927 18:34:16.849215 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:16.849219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:16.849229 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:16.849254 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:16.849255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:16.849268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:16.849330 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:16.849834 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.850496 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.851219 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.851517 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.851574 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.853549 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:16.853555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:16.853556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:16.853562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:16.854919 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.855195 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.855197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.855198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.855246 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.855392 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.855393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.855393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.855537 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:16.855662 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:16.855712 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 18:34:16.855769 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
W20260927 18:34:16.856251 5884 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45910
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:16.875749 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:16.869754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.5.151.193:40053)
0927 18:34:16.869821 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:16.869823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:16.869885 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:34:16.869923 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:16.869924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:16.869934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:16.869983 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:16.870454 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.870935 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.871596 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:16.871776 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:16.871828 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:16.873795 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:16.873797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:16.873797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:16.873803 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:16.874951 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.875252 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.875253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.875253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.875298 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:16.875430 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:16.875430 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:16.875430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:16.875501 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:16.875581 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:16.875637 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:34:16.875692 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:34:16.876197 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 5841
I20260927 18:34:16.881315 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 5975
I20260927 18:34:16.886360 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 6109
I20260927 18:34:16.890596 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 5747
2026-09-27T18:34:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1816 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[6252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[6252](info): set up 2 sockets
Sep 27 18:34:16 dist-test-slave-19nl krb5kdc[6252](info): commencing operation
krb5kdc: starting...
W20260927 18:34:17.889851 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:17 dist-test-slave-19nl krb5kdc[6252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:17.998438 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:40069
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:18.075803 6265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:18.075989 6265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:18.076009 6265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:18.077088 6265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:18.077127 6265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:18.077142 6265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:18.077157 6265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:18.077200 6265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:18.078665 6265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:18.078905 6265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:18.079087 6265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:18.083482 6270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:18.083518 6273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:18.083482 6271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:18.083755 6265 server_base.cc:1034] running on GCE node
I20260927 18:34:18.083999 6265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:18.084285 6265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:18.085422 6265 hybrid_clock.cc:648] HybridClock initialized: now 1790534058085413 us; error 26 us; skew 500 ppm
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:18.087792 6265 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:18.088224 6265 webserver.cc:460] Webserver started at http://127.5.151.254:42039/ using document root <none> and password file <none>
I20260927 18:34:18.088438 6265 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:18.088500 6265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:18.088631 6265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:18.089625 6265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "db73d290f3314b388631149239e85b9e"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "ae0c3b4c4c80e0e38219782554b239fd"
server_key_iv: "6320a6809b52bd3a3ff3ffca38c2a06f"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.089978 6265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "db73d290f3314b388631149239e85b9e"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "ae0c3b4c4c80e0e38219782554b239fd"
server_key_iv: "6320a6809b52bd3a3ff3ffca38c2a06f"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.091341 6265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:18.091980 6280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.092133 6265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:18.092226 6265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "db73d290f3314b388631149239e85b9e"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "ae0c3b4c4c80e0e38219782554b239fd"
server_key_iv: "6320a6809b52bd3a3ff3ffca38c2a06f"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.092303 6265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:18.117905 6265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:18.119823 6265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:18.119982 6265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:18.123883 6265 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:40069
I20260927 18:34:18.123919 6332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:40069 every 8 connection(s)
I20260927 18:34:18.124281 6265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:18.125116 6333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:18.127367 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e: Bootstrap starting.
I20260927 18:34:18.127930 6333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:18.128170 6333 log.cc:824] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:18.128782 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e: No bootstrap required, opened a new log
I20260927 18:34:18.130309 6333 raft_consensus.cc:348] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db73d290f3314b388631149239e85b9e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40069 } }
I20260927 18:34:18.130446 6333 raft_consensus.cc:374] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:18.130470 6333 raft_consensus.cc:729] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db73d290f3314b388631149239e85b9e, State: Initialized, Role: FOLLOWER
I20260927 18:34:18.130637 6333 consensus_queue.cc:260] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db73d290f3314b388631149239e85b9e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40069 } }
I20260927 18:34:18.130715 6333 raft_consensus.cc:388] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:18.130738 6333 raft_consensus.cc:482] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:18.130771 6333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:18.131253 6333 raft_consensus.cc:504] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db73d290f3314b388631149239e85b9e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40069 } }
I20260927 18:34:18.131333 6333 leader_election.cc:302] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db73d290f3314b388631149239e85b9e; no voters:
I20260927 18:34:18.131453 6333 leader_election.cc:288] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:18.131522 6336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:18.131678 6336 raft_consensus.cc:686] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [term 1 LEADER]: Becoming Leader. State: Replica: db73d290f3314b388631149239e85b9e, State: Running, Role: LEADER
I20260927 18:34:18.131716 6333 sys_catalog.cc:564] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:18.131783 6336 consensus_queue.cc:237] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db73d290f3314b388631149239e85b9e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40069 } }
I20260927 18:34:18.132139 6337 sys_catalog.cc:455] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db73d290f3314b388631149239e85b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db73d290f3314b388631149239e85b9e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40069 } } }
I20260927 18:34:18.132283 6337 sys_catalog.cc:458] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:18.132143 6338 sys_catalog.cc:455] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader db73d290f3314b388631149239e85b9e. Latest consensus state: current_term: 1 leader_uuid: "db73d290f3314b388631149239e85b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db73d290f3314b388631149239e85b9e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40069 } } }
I20260927 18:34:18.132441 6338 sys_catalog.cc:458] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:18.132743 6340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:18.132786 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 6265
I20260927 18:34:18.132881 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:18.133041 5727 external_mini_cluster.cc:1428] Setting key 8426116666aacac9a833520f7e9813d7
I20260927 18:34:18.133678 6340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:18.135308 6340 catalog_manager.cc:1347] Generated new cluster ID: 77aafaf9dddb4a2dbe6a5c5abe17778e
I20260927 18:34:18.135377 6340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:18.143134 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.134319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.5.151.254:40069)
0927 18:34:18.134490 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.134498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.134529 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:34:18.134660 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.134667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.134721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.134868 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:18.135526 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.136421 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.137177 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.137625 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.137754 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.140056 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:18.140060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:18.140061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:18.140073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:18.141678 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.142034 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.142037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.142038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.142132 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.142330 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.142331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.142332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.142594 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:18.142777 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.142842 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.142971 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260927 18:34:18.150998 6340 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:18.151469 6340 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:18.156961 6340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db73d290f3314b388631149239e85b9e: Generated new TSK 0
I20260927 18:34:18.157122 6340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:18.189046 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40069
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:18.268743 6359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:18.268936 6359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:18.268958 6359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:18.270043 6359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:18.270079 6359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:18.270107 6359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:18.271543 6359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:18.271729 6359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:18.271903 6359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:18.276561 6364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:18.276584 6367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:18.276584 6365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:18.276768 6359 server_base.cc:1034] running on GCE node
I20260927 18:34:18.277014 6359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:18.277242 6359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:18.278394 6359 hybrid_clock.cc:648] HybridClock initialized: now 1790534058278377 us; error 29 us; skew 500 ppm
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:18.280896 6359 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:18.281311 6359 webserver.cc:460] Webserver started at http://127.5.151.193:37613/ using document root <none> and password file <none>
I20260927 18:34:18.281517 6359 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:18.281560 6359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:18.281713 6359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:18.282771 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "0c265e1bfc914c6bb110b8a1249a93f6"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "a1e386127a66788115c536c9ce956f6f"
server_key_iv: "f68be462d0ee3aa12ea9186c59582e79"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.283108 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "0c265e1bfc914c6bb110b8a1249a93f6"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "a1e386127a66788115c536c9ce956f6f"
server_key_iv: "f68be462d0ee3aa12ea9186c59582e79"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.284341 6359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:18.284991 6374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.285151 6359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:18.285225 6359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "0c265e1bfc914c6bb110b8a1249a93f6"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "a1e386127a66788115c536c9ce956f6f"
server_key_iv: "f68be462d0ee3aa12ea9186c59582e79"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.285280 6359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:18.293265 6359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:18.295181 6359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:18.295323 6359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:18.295521 6359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:18.295804 6359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:18.295861 6359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.295888 6359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:18.295912 6359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.301239 6359 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:45587
I20260927 18:34:18.301302 6487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:45587 every 8 connection(s)
I20260927 18:34:18.301589 6359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:18.303563 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 6359
I20260927 18:34:18.303658 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:18.303823 5727 external_mini_cluster.cc:1428] Setting key 8bc9ac38504c52ab3fef1ce3e4bf4545
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:18.311339 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.302261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:48573 (local address 127.5.151.254:40069)
0927 18:34:18.302322 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.302324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.302524 (+ 200us) server_negotiation.cc:407] Connection header received
0927 18:34:18.303492 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.303493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.303510 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.303561 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:18.304302 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.304844 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.305822 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.306034 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.306112 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.308510 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:18.308511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:18.308511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:18.308517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:18.309911 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.310382 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.310383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.310383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.310433 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.310716 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.310716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.310717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.310826 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:18.310879 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.311199 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.311275 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:34:18.311601 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.302249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:40069 (local address 127.5.151.193:48573)
0927 18:34:18.302414 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.302451 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.303378 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.303626 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.303630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.303984 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:18.304206 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.304210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.304930 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.304932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.305715 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.305717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.306122 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.306123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.306249 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.306764 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:18.306772 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:18.308419 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:18.309991 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:18.309998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:18.310000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:18.310307 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:18.310497 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:18.310498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:18.310498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:18.310652 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:18.310894 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:18.310900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:18.311059 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.311172 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.311336 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
I20260927 18:34:18.312114 6488 heartbeater.cc:347] Connected to a master server at 127.5.151.254:40069
I20260927 18:34:18.312212 6488 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:18.312381 6488 heartbeater.cc:511] Master 127.5.151.254:40069 requested a full tablet report, sending...
I20260927 18:34:18.312803 6297 ts_manager.cc:194] Registered new tserver with Master: 0c265e1bfc914c6bb110b8a1249a93f6 (127.5.151.193:45587)
I20260927 18:34:18.313455 6297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:48573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:18.327148 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40069
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:18.404887 6493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:18.405072 6493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:18.405094 6493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:18.406157 6493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:18.406194 6493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:18.406219 6493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:18.407588 6493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:18.407763 6493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:18.407941 6493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:18.412359 6499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:18.412513 6493 server_base.cc:1034] running on GCE node
W20260927 18:34:18.412371 6498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:18.412384 6501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:18.412846 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:18.413081 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:18.414227 6493 hybrid_clock.cc:648] HybridClock initialized: now 1790534058414232 us; error 41 us; skew 500 ppm
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:18.416621 6493 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:18.417006 6493 webserver.cc:460] Webserver started at http://127.5.151.194:45719/ using document root <none> and password file <none>
I20260927 18:34:18.417238 6493 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:18.417305 6493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:18.417438 6493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:18.418351 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "b3a917e6bb3f429198585e0b09f61d0c"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "f2d865d54d6c88ded3ab5edca529d169"
server_key_iv: "66efdcbe18ff84d91437303a9996929a"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.418709 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "b3a917e6bb3f429198585e0b09f61d0c"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "f2d865d54d6c88ded3ab5edca529d169"
server_key_iv: "66efdcbe18ff84d91437303a9996929a"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.419919 6493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:18.420538 6508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.420684 6493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:18.420786 6493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "b3a917e6bb3f429198585e0b09f61d0c"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "f2d865d54d6c88ded3ab5edca529d169"
server_key_iv: "66efdcbe18ff84d91437303a9996929a"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.420838 6493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:18.444955 6493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:18.446762 6493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:18.446869 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:18.447036 6493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:18.447353 6493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:18.447389 6493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.447413 6493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:18.447425 6493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.452816 6493 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:41441
I20260927 18:34:18.452888 6621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:41441 every 8 connection(s)
I20260927 18:34:18.453198 6493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:18.461324 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 6493
I20260927 18:34:18.461417 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:18.461599 5727 external_mini_cluster.cc:1428] Setting key d8f24fff6746a2f4f98174f68f03fb43
I20260927 18:34:18.461668 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.453904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:59967 (local address 127.5.151.254:40069)
0927 18:34:18.453953 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.453955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.454140 (+ 185us) server_negotiation.cc:407] Connection header received
0927 18:34:18.455125 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.455126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.455136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.455193 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:18.455741 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.456220 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.457136 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.457318 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.457367 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.459348 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:18.459349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:18.459350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:18.459355 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:18.460488 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.460830 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.460830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.460831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.460876 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.461063 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.461064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.461064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.461157 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:18.461206 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.461519 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.461583 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:34:18.461910 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.453867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:40069 (local address 127.5.151.194:59967)
0927 18:34:18.454019 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.454060 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.455033 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.455248 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.455251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.455489 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:18.455674 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.455677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.456291 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.456292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.457070 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.457071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.457381 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.457382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.457478 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.457911 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:18.457919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:18.459252 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:18.460555 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:18.460560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:18.460562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:18.460764 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:18.460928 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:18.460928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:18.460928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:18.461016 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:18.461216 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:18.461221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:18.461387 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.461505 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.461669 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260927 18:34:18.462404 6622 heartbeater.cc:347] Connected to a master server at 127.5.151.254:40069
I20260927 18:34:18.462500 6622 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:18.462707 6622 heartbeater.cc:511] Master 127.5.151.254:40069 requested a full tablet report, sending...
I20260927 18:34:18.463099 6296 ts_manager.cc:194] Registered new tserver with Master: b3a917e6bb3f429198585e0b09f61d0c (127.5.151.194:41441)
I20260927 18:34:18.463477 6296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:59967
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:18.484297 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40069
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:18.563179 6627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:18.563386 6627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:18.563410 6627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:18.564496 6627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:18.564529 6627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:18.564555 6627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:18.565925 6627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:18.566108 6627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:18.566273 6627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:18.570906 6633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:18.570916 6635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:18.570924 6632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:18.571116 6627 server_base.cc:1034] running on GCE node
I20260927 18:34:18.571311 6627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:18.571484 6627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:18.572643 6627 hybrid_clock.cc:648] HybridClock initialized: now 1790534058572632 us; error 35 us; skew 500 ppm
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:18.578480 6627 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:18.578940 6627 webserver.cc:460] Webserver started at http://127.5.151.195:42103/ using document root <none> and password file <none>
I20260927 18:34:18.579118 6627 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:18.579157 6627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:18.579244 6627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:18.580423 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "d75fa8b3e37f44a9819dc00488d3b857"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "2f6906ff3e7856dcdee00e12003a9753"
server_key_iv: "61a8d682bf38fe8b386bb557a18178f5"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.580715 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "d75fa8b3e37f44a9819dc00488d3b857"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "2f6906ff3e7856dcdee00e12003a9753"
server_key_iv: "61a8d682bf38fe8b386bb557a18178f5"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.581961 6627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:18.582710 6642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.582865 6627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:18.582943 6627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "d75fa8b3e37f44a9819dc00488d3b857"
format_stamp: "Formatted at 2026-09-27 18:34:18 on dist-test-slave-19nl"
server_key: "2f6906ff3e7856dcdee00e12003a9753"
server_key_iv: "61a8d682bf38fe8b386bb557a18178f5"
server_key_version: "encryptionkey@0"
I20260927 18:34:18.582994 6627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:18.590759 6627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:18.593027 6627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:18.593145 6627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:18.593338 6627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:18.593653 6627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:18.593688 6627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.593734 6627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:18.593768 6627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:18.599654 6627 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:43349
I20260927 18:34:18.599723 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:43349 every 8 connection(s)
I20260927 18:34:18.600018 6627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:18.608362 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 6627
I20260927 18:34:18.608470 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:18.608650 5727 external_mini_cluster.cc:1428] Setting key 05432cd514527cf6f4ca24382a10bd79
I20260927 18:34:18.609040 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.600722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:49909 (local address 127.5.151.254:40069)
0927 18:34:18.600767 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.600769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.600948 (+ 179us) server_negotiation.cc:407] Connection header received
0927 18:34:18.601913 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.601915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.601928 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.601974 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:18.602609 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.603107 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.604102 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.604304 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.604387 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.606346 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:18.606347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:18.606348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:18.606355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:18.607705 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.608047 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.608047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.608048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.608095 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.608298 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.608300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.608300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.608452 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:18.608531 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.608845 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.608931 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:34:18.609292 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.600689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:40069 (local address 127.5.151.195:49909)
0927 18:34:18.600830 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.600868 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.601799 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.602054 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.602058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.602311 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:18.602499 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.602503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.603213 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.603214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.604013 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.604015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.604395 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.604396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.604517 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.604961 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:18.604969 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:18.606249 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:18.607768 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:18.607773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:18.607775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:18.607974 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:18.608146 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:18.608147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:18.608147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:18.608236 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:18.608533 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:18.608538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:18.608672 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.608821 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.609012 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260927 18:34:18.609705 6756 heartbeater.cc:347] Connected to a master server at 127.5.151.254:40069
I20260927 18:34:18.609812 6756 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:18.609972 6756 heartbeater.cc:511] Master 127.5.151.254:40069 requested a full tablet report, sending...
I20260927 18:34:18.610316 6296 ts_manager.cc:194] Registered new tserver with Master: d75fa8b3e37f44a9819dc00488d3b857 (127.5.151.195:43349)
I20260927 18:34:18.610741 6296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:49909
I20260927 18:34:18.620537 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:34:18.627894 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.622309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.5.151.254:40069)
0927 18:34:18.622362 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.622364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.622475 (+ 111us) server_negotiation.cc:407] Connection header received
0927 18:34:18.622514 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.622515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.622523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.622629 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:18.623202 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.623656 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.624397 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.624563 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.624612 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.625652 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:18.625653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:18.625653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:18.625658 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:18.626994 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.627257 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.627258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.627258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.627308 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.627444 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.627444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.627445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.627565 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:18.627683 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.627755 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.627816 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:34:18.630195 6296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:18.631799 6296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:18.638084 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.634266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.5.151.194:41441)
0927 18:34:18.634458 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.634463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.634471 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:18.634506 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.634509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.634525 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.634658 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:18.634776 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.635705 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.637125 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.637662 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.637754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.637816 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.637963 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.638004 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
I20260927 18:34:18.638077 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.634024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:41441 (local address 127.0.0.1:56868)
0927 18:34:18.634338 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.634346 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.634394 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.634658 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.634659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.634661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:18.634718 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.634720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.635805 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.635806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.637031 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.637034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.637757 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.637758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.637834 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.637846 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.637946 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.638000 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":55,"threads_started":1}
I20260927 18:34:18.638195 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.634219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:43349 (local address 127.0.0.1:59048)
0927 18:34:18.634597 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.634607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.634663 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.634876 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.634876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.634878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:18.634907 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.634908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.636167 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.636168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.636997 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.637000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.637935 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.637936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.638001 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.638006 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.638051 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.638095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":48,"threads_started":1}
I20260927 18:34:18.638476 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.634403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.5.151.195:43349)
0927 18:34:18.634609 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.634613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.634722 (+ 109us) server_negotiation.cc:407] Connection header received
0927 18:34:18.634766 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.634768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.634783 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.634872 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:18.634965 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.636055 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.637086 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.637764 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.637844 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.637897 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.638356 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.638404 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
I20260927 18:34:18.638656 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.634021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:45587 (local address 127.0.0.1:32786)
0927 18:34:18.634128 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.634167 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.634233 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.634599 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.634602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.634605 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:18.634676 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.634677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.636146 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.636147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.637184 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.637185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.638085 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.638086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.638371 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.638376 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.638423 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.638459 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":63,"threads_started":1}
I20260927 18:34:18.638886 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.634122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.5.151.193:45587)
0927 18:34:18.634319 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.634324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.634336 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:34:18.634377 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.634380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.634397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.634481 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:18.634761 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.636039 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.637257 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.637948 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.638007 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.638056 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.638755 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.638804 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260927 18:34:18.638849 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 70b23b8f1dfa459d8d8ae3d67fa3cacd (DEFAULT_TABLE table=test-table [id=3a922f252c5e41578256de94a0ffa466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:18.639075 6690 tablet_service.cc:1467] Processing CreateTablet for tablet 70b23b8f1dfa459d8d8ae3d67fa3cacd (DEFAULT_TABLE table=test-table [id=3a922f252c5e41578256de94a0ffa466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:18.639192 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b23b8f1dfa459d8d8ae3d67fa3cacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:18.639344 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b23b8f1dfa459d8d8ae3d67fa3cacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:18.639418 6422 tablet_service.cc:1467] Processing CreateTablet for tablet 70b23b8f1dfa459d8d8ae3d67fa3cacd (DEFAULT_TABLE table=test-table [id=3a922f252c5e41578256de94a0ffa466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:18.639652 6422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b23b8f1dfa459d8d8ae3d67fa3cacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:18.642416 6771 tablet_bootstrap.cc:492] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c: Bootstrap starting.
I20260927 18:34:18.642414 6772 tablet_bootstrap.cc:492] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857: Bootstrap starting.
I20260927 18:34:18.642659 6770 tablet_bootstrap.cc:492] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6: Bootstrap starting.
I20260927 18:34:18.643090 6771 tablet_bootstrap.cc:654] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:18.643090 6772 tablet_bootstrap.cc:654] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:18.643392 6770 tablet_bootstrap.cc:654] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:18.643674 6772 log.cc:824] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:18.643674 6771 log.cc:824] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:18.643922 6770 log.cc:824] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:18.644431 6772 tablet_bootstrap.cc:492] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857: No bootstrap required, opened a new log
I20260927 18:34:18.644501 6772 ts_tablet_manager.cc:1397] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:18.644595 6771 tablet_bootstrap.cc:492] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c: No bootstrap required, opened a new log
I20260927 18:34:18.644649 6771 ts_tablet_manager.cc:1397] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:18.646464 6772 raft_consensus.cc:348] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.646464 6771 raft_consensus.cc:348] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.646674 6772 raft_consensus.cc:374] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:18.646674 6771 raft_consensus.cc:374] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:18.646720 6772 raft_consensus.cc:729] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75fa8b3e37f44a9819dc00488d3b857, State: Initialized, Role: FOLLOWER
I20260927 18:34:18.646720 6771 raft_consensus.cc:729] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a917e6bb3f429198585e0b09f61d0c, State: Initialized, Role: FOLLOWER
I20260927 18:34:18.646859 6772 consensus_queue.cc:260] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.646848 6771 consensus_queue.cc:260] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.647140 6771 ts_tablet_manager.cc:1428] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:34:18.647140 6772 ts_tablet_manager.cc:1428] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:34:18.647166 6770 tablet_bootstrap.cc:492] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6: No bootstrap required, opened a new log
I20260927 18:34:18.647223 6770 ts_tablet_manager.cc:1397] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 18:34:18.647279 6622 heartbeater.cc:503] Master 127.5.151.254:40069 was elected leader, sending a full tablet report...
I20260927 18:34:18.647305 6756 heartbeater.cc:503] Master 127.5.151.254:40069 was elected leader, sending a full tablet report...
I20260927 18:34:18.649055 6770 raft_consensus.cc:348] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.649205 6770 raft_consensus.cc:374] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:18.649228 6770 raft_consensus.cc:729] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c265e1bfc914c6bb110b8a1249a93f6, State: Initialized, Role: FOLLOWER
I20260927 18:34:18.649330 6770 consensus_queue.cc:260] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.649541 6770 ts_tablet_manager.cc:1428] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:34:18.649586 6488 heartbeater.cc:503] Master 127.5.151.254:40069 was elected leader, sending a full tablet report...
W20260927 18:34:18.703791 6623 tablet.cc:2369] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:18.802273 6489 tablet.cc:2369] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:18.850693 6757 tablet.cc:2369] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:18.925623 6777 raft_consensus.cc:482] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:18.925747 6777 raft_consensus.cc:504] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.926048 6777 leader_election.cc:288] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c265e1bfc914c6bb110b8a1249a93f6 (127.5.151.193:45587), d75fa8b3e37f44a9819dc00488d3b857 (127.5.151.195:43349)
I20260927 18:34:18.929072 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.926342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:50219 (local address 127.5.151.193:45587)
0927 18:34:18.926413 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.926415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.926428 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:34:18.926491 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.926492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.926505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.926601 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:18.926784 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.927318 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.928361 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.928713 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.928768 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.928775 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.928951 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.928993 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:34:18.929090 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.926490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:46363 (local address 127.5.151.195:43349)
0927 18:34:18.926533 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.926536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.926589 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:34:18.926683 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.926684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.926693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.926754 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:18.926931 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.927424 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.928373 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.928737 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.928824 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.928831 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.928991 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.929029 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:18.929088 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.926274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:45587 (local address 127.5.151.194:50219)
0927 18:34:18.926340 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.926355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.926421 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.926612 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.926613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.926616 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:18.926707 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.926709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.927418 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.927419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.928251 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.928253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.928790 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.928791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.928881 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.928884 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.928926 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.928972 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":29}
I20260927 18:34:18.929109 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.926397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:43349 (local address 127.5.151.194:46363)
0927 18:34:18.926515 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:18.926524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:18.926592 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:18.926776 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:18.926777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:18.926779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:18.926858 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.926860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.927502 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.927503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.928260 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:18.928262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.928848 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.928849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:18.928904 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.928907 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:18.928978 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:34:18.929015 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260927 18:34:18.929486 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b23b8f1dfa459d8d8ae3d67fa3cacd" candidate_uuid: "b3a917e6bb3f429198585e0b09f61d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75fa8b3e37f44a9819dc00488d3b857" is_pre_election: true
I20260927 18:34:18.929541 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b23b8f1dfa459d8d8ae3d67fa3cacd" candidate_uuid: "b3a917e6bb3f429198585e0b09f61d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" is_pre_election: true
I20260927 18:34:18.929657 6710 raft_consensus.cc:2454] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3a917e6bb3f429198585e0b09f61d0c in term 0.
I20260927 18:34:18.929730 6442 raft_consensus.cc:2454] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3a917e6bb3f429198585e0b09f61d0c in term 0.
I20260927 18:34:18.929891 6510 leader_election.cc:302] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3a917e6bb3f429198585e0b09f61d0c, d75fa8b3e37f44a9819dc00488d3b857; no voters:
I20260927 18:34:18.930007 6777 raft_consensus.cc:2781] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:18.930059 6777 raft_consensus.cc:482] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:18.930076 6777 raft_consensus.cc:3037] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:18.930809 6777 raft_consensus.cc:504] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.930939 6777 leader_election.cc:288] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [CANDIDATE]: Term 1 election: Requested vote from peers 0c265e1bfc914c6bb110b8a1249a93f6 (127.5.151.193:45587), d75fa8b3e37f44a9819dc00488d3b857 (127.5.151.195:43349)
I20260927 18:34:18.931134 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b23b8f1dfa459d8d8ae3d67fa3cacd" candidate_uuid: "b3a917e6bb3f429198585e0b09f61d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c265e1bfc914c6bb110b8a1249a93f6"
I20260927 18:34:18.931134 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b23b8f1dfa459d8d8ae3d67fa3cacd" candidate_uuid: "b3a917e6bb3f429198585e0b09f61d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75fa8b3e37f44a9819dc00488d3b857"
I20260927 18:34:18.931213 6442 raft_consensus.cc:3037] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:18.931213 6710 raft_consensus.cc:3037] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:18.932092 6710 raft_consensus.cc:2454] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P d75fa8b3e37f44a9819dc00488d3b857 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3a917e6bb3f429198585e0b09f61d0c in term 1.
I20260927 18:34:18.932092 6442 raft_consensus.cc:2454] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P 0c265e1bfc914c6bb110b8a1249a93f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3a917e6bb3f429198585e0b09f61d0c in term 1.
I20260927 18:34:18.932263 6510 leader_election.cc:302] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3a917e6bb3f429198585e0b09f61d0c, d75fa8b3e37f44a9819dc00488d3b857; no voters:
I20260927 18:34:18.932402 6777 raft_consensus.cc:2781] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:18.932549 6777 raft_consensus.cc:686] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [term 1 LEADER]: Becoming Leader. State: Replica: b3a917e6bb3f429198585e0b09f61d0c, State: Running, Role: LEADER
I20260927 18:34:18.932646 6777 consensus_queue.cc:237] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } }
I20260927 18:34:18.933476 6295 catalog_manager.cc:5495] T 70b23b8f1dfa459d8d8ae3d67fa3cacd P b3a917e6bb3f429198585e0b09f61d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to b3a917e6bb3f429198585e0b09f61d0c (127.5.151.194). New cstate: current_term: 1 leader_uuid: "b3a917e6bb3f429198585e0b09f61d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c265e1bfc914c6bb110b8a1249a93f6" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 45587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d75fa8b3e37f44a9819dc00488d3b857" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3a917e6bb3f429198585e0b09f61d0c" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41441 } health_report { overall_health: HEALTHY } } }
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:18.995949 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:18.988566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.5.151.193:45587)
0927 18:34:18.988644 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:18.988647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:18.988752 (+ 105us) server_negotiation.cc:407] Connection header received
0927 18:34:18.988826 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:18.988828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:18.988839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:18.988893 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:18.989488 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.989976 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.990728 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:18.990927 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:18.990998 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:18.993269 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:18.993287 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:18.993287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:18.993293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:18.994791 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.995141 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.995145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.995145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.995203 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:18.995409 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:18.995410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:18.995410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:18.995549 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:18.995688 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:18.995759 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:34:18.995841 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 18:34:18 dist-test-slave-19nl krb5kdc[6252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:34:19 dist-test-slave-19nl krb5kdc[6252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:19.018172 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:19.011343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.5.151.193:45587)
0927 18:34:19.011416 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:19.011418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:19.011521 (+ 103us) server_negotiation.cc:407] Connection header received
0927 18:34:19.011579 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:19.011580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:19.011590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:19.011639 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:19.012165 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:19.012652 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:19.013378 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:19.013552 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:19.013602 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:19.015798 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:19.015799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:19.015799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:19.015806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:19.017181 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:19.017492 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:19.017493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:19.017494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:19.017542 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:19.017695 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:19.017696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:19.017696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:19.017781 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:19.017854 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:19.017997 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 18:34:19.018070 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 18:34:19 dist-test-slave-19nl krb5kdc[6252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:34:19 dist-test-slave-19nl krb5kdc[6252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:19.039630 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:19.032902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.5.151.193:45587)
0927 18:34:19.032967 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:19.032969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:19.033053 (+ 84us) server_negotiation.cc:407] Connection header received
0927 18:34:19.033161 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:19.033162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:19.033172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:19.033217 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:19.033757 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:19.034217 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:19.035022 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:19.035214 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:19.035291 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:19.037351 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:19.037352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:19.037352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:19.037359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:19.038507 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:19.038888 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:19.038889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:19.038890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:19.038945 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:19.039141 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:19.039142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:19.039143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:19.039237 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:19.039342 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:19.039434 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 18:34:19.039527 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 18:34:19.040097 6402 tablet_service.cc:2908] Rejecting scan request for tablet 70b23b8f1dfa459d8d8ae3d67fa3cacd: Uninitialized: clean time has not yet been initialized
I20260927 18:34:19.040868 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 6359
I20260927 18:34:19.045410 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 6493
I20260927 18:34:19.049594 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 6627
I20260927 18:34:19.053604 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 6265
2026-09-27T18:34:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2164 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:19 dist-test-slave-19nl krb5kdc[6794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:19 dist-test-slave-19nl krb5kdc[6794](info): set up 2 sockets
Sep 27 18:34:19 dist-test-slave-19nl krb5kdc[6794](info): commencing operation
krb5kdc: starting...
W20260927 18:34:20.062076 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:20.170111 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41597
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:20.248694 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:20.248900 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:20.248922 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:20.250033 6807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:20.250066 6807 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:34:20.250078 6807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:20.250092 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:20.250102 6807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:20.250114 6807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:20.251572 6807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41597
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:20.251754 6807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:20.251911 6807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:20.256703 6812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:20.256738 6813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:20.256748 6807 server_base.cc:1034] running on GCE node
W20260927 18:34:20.256716 6815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:20.257153 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:20.257455 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:20.258610 6807 hybrid_clock.cc:648] HybridClock initialized: now 1790534060258591 us; error 35 us; skew 500 ppm
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:20.265599 6807 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:20.266028 6807 webserver.cc:460] Webserver started at http://127.5.151.254:35893/ using document root <none> and password file <none>
I20260927 18:34:20.266244 6807 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:20.266310 6807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:20.266449 6807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:20.267526 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "ad72486fd8464142b7548f36dbb5d6c5"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "7db13d3826fa1ac704afda5824f1e944"
server_key_iv: "19ed3fe523162b1686742a703e924a8b"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.267863 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "ad72486fd8464142b7548f36dbb5d6c5"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "7db13d3826fa1ac704afda5824f1e944"
server_key_iv: "19ed3fe523162b1686742a703e924a8b"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.269192 6807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:20.269924 6822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:20.270078 6807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:20.270175 6807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "ad72486fd8464142b7548f36dbb5d6c5"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "7db13d3826fa1ac704afda5824f1e944"
server_key_iv: "19ed3fe523162b1686742a703e924a8b"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.270241 6807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:20.288820 6807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:20.290644 6807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:20.290797 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:20.295288 6807 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:41597
I20260927 18:34:20.295346 6884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:41597 every 8 connection(s)
I20260927 18:34:20.295652 6807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:20.296236 6885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:20.298415 6885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5: Bootstrap starting.
I20260927 18:34:20.298997 6885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:20.299283 6885 log.cc:824] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:20.299908 6885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5: No bootstrap required, opened a new log
I20260927 18:34:20.301913 6885 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad72486fd8464142b7548f36dbb5d6c5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41597 } }
I20260927 18:34:20.302049 6885 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:20.302073 6885 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad72486fd8464142b7548f36dbb5d6c5, State: Initialized, Role: FOLLOWER
I20260927 18:34:20.302237 6885 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad72486fd8464142b7548f36dbb5d6c5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41597 } }
I20260927 18:34:20.302299 6885 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:20.302320 6885 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:20.302383 6885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:20.303001 6885 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad72486fd8464142b7548f36dbb5d6c5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41597 } }
I20260927 18:34:20.303120 6885 leader_election.cc:302] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad72486fd8464142b7548f36dbb5d6c5; no voters:
I20260927 18:34:20.303282 6885 leader_election.cc:288] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:20.303345 6888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:20.303503 6888 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [term 1 LEADER]: Becoming Leader. State: Replica: ad72486fd8464142b7548f36dbb5d6c5, State: Running, Role: LEADER
I20260927 18:34:20.303535 6885 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:20.303591 6888 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad72486fd8464142b7548f36dbb5d6c5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41597 } }
I20260927 18:34:20.303952 6889 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad72486fd8464142b7548f36dbb5d6c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad72486fd8464142b7548f36dbb5d6c5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41597 } } }
I20260927 18:34:20.303978 6890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad72486fd8464142b7548f36dbb5d6c5. Latest consensus state: current_term: 1 leader_uuid: "ad72486fd8464142b7548f36dbb5d6c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad72486fd8464142b7548f36dbb5d6c5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41597 } } }
I20260927 18:34:20.304157 6890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:20.304526 6889 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:20.304550 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 6807
I20260927 18:34:20.304539 6897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:20.304641 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:20.304817 5727 external_mini_cluster.cc:1428] Setting key 579b17120cd030ed2e85f0720edbc36e
I20260927 18:34:20.305135 6897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:20.306703 6897 catalog_manager.cc:1347] Generated new cluster ID: 99d95489df104ccb975dc6e693447201
I20260927 18:34:20.306751 6897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:20.319171 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.305397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.5.151.254:41597)
0927 18:34:20.305882 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.305890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.305919 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:34:20.306026 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.306033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.306104 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.306277 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.311022 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.312294 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.313045 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.313483 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.313604 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.315913 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.315917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.315917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.315933 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.317534 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.317904 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.317907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.317907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.318004 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.318201 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.318202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.318202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.318440 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.318641 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.318748 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.318983 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":77,"threads_started":1}
I20260927 18:34:20.342291 6897 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:20.342891 6897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:20.350283 6897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5: Generated new TSK 0
I20260927 18:34:20.350488 6897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:20.378476 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41597
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:20.456594 6911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:20.456786 6911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:20.456821 6911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:20.457897 6911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:20.457958 6911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:20.457991 6911 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:20.458025 6911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:20.459417 6911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:41597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:20.459676 6911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:20.459888 6911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:20.464722 6917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:20.464746 6916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:20.464737 6919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:20.464754 6911 server_base.cc:1034] running on GCE node
I20260927 18:34:20.465109 6911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:20.465332 6911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:20.466465 6911 hybrid_clock.cc:648] HybridClock initialized: now 1790534060466455 us; error 29 us; skew 500 ppm
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:20.468858 6911 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:20.469276 6911 webserver.cc:460] Webserver started at http://127.5.151.193:35055/ using document root <none> and password file <none>
I20260927 18:34:20.469499 6911 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:20.469544 6911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:20.469668 6911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:20.470719 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "1100c997f4b04333a9ea283aa6cf83c4"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "03fcefaa8d1d89d858e91a1a41f070c0"
server_key_iv: "294997b22a84819ad0d6d3906e417b61"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.471047 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "1100c997f4b04333a9ea283aa6cf83c4"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "03fcefaa8d1d89d858e91a1a41f070c0"
server_key_iv: "294997b22a84819ad0d6d3906e417b61"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.472286 6911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:20.473039 6926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:20.473193 6911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:20.473284 6911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "1100c997f4b04333a9ea283aa6cf83c4"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "03fcefaa8d1d89d858e91a1a41f070c0"
server_key_iv: "294997b22a84819ad0d6d3906e417b61"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.473353 6911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:20.485668 6911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:20.487785 6911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:20.487936 6911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:20.488436 6911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:20.488489 6911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:20.488529 6911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:20.488552 6911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:20.495878 6911 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:39321
I20260927 18:34:20.495954 7041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:39321 every 8 connection(s)
I20260927 18:34:20.496299 6911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:20.498299 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.488782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:43003 (local address 127.5.151.254:41597)
0927 18:34:20.488846 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.488848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.489107 (+ 259us) server_negotiation.cc:407] Connection header received
0927 18:34:20.489815 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.489816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.489829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.489871 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.490767 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.491277 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.492927 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.493168 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.493251 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.495866 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.495868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.495868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.495876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.497244 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.497577 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.497578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.497578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.497627 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.497830 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.497831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.497831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.497927 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.497976 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.498176 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.498238 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:34:20.498451 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.488774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41597 (local address 127.5.151.193:43003)
0927 18:34:20.488961 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.489008 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.489714 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.489947 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.489951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.490329 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:20.490653 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.490660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.491346 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.491347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.492819 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.492822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.493268 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.493269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.493425 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.493944 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:20.493953 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:20.495753 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:20.497307 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.497310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.497312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.497512 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.497677 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.497678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.497678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.497764 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.497975 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:20.497979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:20.498087 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.498165 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.498274 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260927 18:34:20.498903 7042 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41597
I20260927 18:34:20.498982 7042 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:20.499151 7042 heartbeater.cc:511] Master 127.5.151.254:41597 requested a full tablet report, sending...
I20260927 18:34:20.499655 6838 ts_manager.cc:194] Registered new tserver with Master: 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193:39321)
I20260927 18:34:20.500314 6838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:43003
I20260927 18:34:20.503180 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 6911
I20260927 18:34:20.503298 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:20.503738 5727 external_mini_cluster.cc:1428] Setting key 29d6c580a737a3f272c330306bda5aea
I20260927 18:34:20.505795 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.499922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:57407 (local address 127.5.151.254:41597)
0927 18:34:20.500505 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.500507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.500515 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:20.500534 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.500534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.500543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.500601 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.500917 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.501442 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.502089 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.502258 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.502305 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.503203 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.503204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.503204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.503209 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.504700 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.505025 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.505026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.505026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.505117 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.505288 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.505289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.505290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.505394 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.505464 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.505627 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.505720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566}
I20260927 18:34:20.505827 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.499866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41597 (local address 127.5.151.193:57407)
0927 18:34:20.499969 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.499979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.500030 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.500615 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.500616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.500797 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:20.500843 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.500845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.501523 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.501524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.502020 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.502022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.502314 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.502314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.502367 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.502807 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:20.502811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:20.503129 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:20.504781 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.504782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.504783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.504945 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.505167 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.505168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.505168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.505223 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.505455 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:20.505456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:20.505499 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.505576 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.505692 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:20.524676 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41597
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:20.603044 7051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:20.603251 7051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:20.603289 7051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:20.604586 7051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:20.604643 7051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:20.604679 7051 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:20.604719 7051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:20.606081 7051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:41597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:20.606315 7051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:20.606506 7051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:20.611063 7056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:20.611073 7059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:20.611073 7057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:20.611339 7051 server_base.cc:1034] running on GCE node
I20260927 18:34:20.611550 7051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:20.611788 7051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:20.612934 7051 hybrid_clock.cc:648] HybridClock initialized: now 1790534060612930 us; error 36 us; skew 500 ppm
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:20.615228 7051 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:20.615675 7051 webserver.cc:460] Webserver started at http://127.5.151.194:32931/ using document root <none> and password file <none>
I20260927 18:34:20.615908 7051 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:20.615968 7051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:20.616110 7051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:20.617055 7051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "9b4fb3b01fc940f9b77efe7a47412b2a"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "3c39ab1e2acc823db6842003545753c3"
server_key_iv: "56038550d4c2e9449b3b611ba77f9559"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.617393 7051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "9b4fb3b01fc940f9b77efe7a47412b2a"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "3c39ab1e2acc823db6842003545753c3"
server_key_iv: "56038550d4c2e9449b3b611ba77f9559"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.618665 7051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:20.619354 7066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:20.619547 7051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:20.619635 7051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "9b4fb3b01fc940f9b77efe7a47412b2a"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "3c39ab1e2acc823db6842003545753c3"
server_key_iv: "56038550d4c2e9449b3b611ba77f9559"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.619711 7051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:20.644212 7051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:20.645959 7051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:20.646116 7051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:20.646689 7051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:20.646744 7051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:20.646785 7051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:20.646813 7051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:20.653811 7051 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:37227
I20260927 18:34:20.653853 7181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:37227 every 8 connection(s)
I20260927 18:34:20.654135 7051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:20.655499 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.647069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:47041 (local address 127.5.151.254:41597)
0927 18:34:20.647107 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.647109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.647331 (+ 222us) server_negotiation.cc:407] Connection header received
0927 18:34:20.647866 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.647868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.647879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.647940 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.648526 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.649072 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.649985 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.650177 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.650270 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.652857 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.652858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.652859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.652866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.654429 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.654827 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.654828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.654829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.654880 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.655057 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.655057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.655057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.655155 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.655202 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.655377 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.655436 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:34:20.655686 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.647075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41597 (local address 127.5.151.194:47041)
0927 18:34:20.647225 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.647259 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.647769 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.648019 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.648023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.648248 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:20.648427 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.648432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.649153 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.649154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.649901 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.649903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.650280 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.650282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.650398 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.650901 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:20.650908 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:20.652753 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:20.654519 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.654524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.654525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.654766 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.654930 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.654931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.654931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.655010 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.655191 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:20.655193 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:20.655288 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.655354 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.655455 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260927 18:34:20.656157 7182 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41597
I20260927 18:34:20.656236 7182 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:20.656416 7182 heartbeater.cc:511] Master 127.5.151.254:41597 requested a full tablet report, sending...
I20260927 18:34:20.656747 6839 ts_manager.cc:194] Registered new tserver with Master: 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194:37227)
I20260927 18:34:20.657226 6839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:47041
I20260927 18:34:20.658962 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 7051
I20260927 18:34:20.659049 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:20.659232 5727 external_mini_cluster.cc:1428] Setting key 1613813400e6a8179cae0a297e7d79e9
I20260927 18:34:20.662436 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.657057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41597 (local address 127.5.151.194:45611)
0927 18:34:20.657185 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.657196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.657256 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.657393 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.657394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.657557 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:20.657613 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.657614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.658181 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.658182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.658717 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.658719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.659116 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.659117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.659198 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.659624 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:20.659628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:20.659967 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:20.661587 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.661588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.661588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.661728 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.661909 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.661910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.661910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.661955 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.662240 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:20.662241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:20.662271 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.662313 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.662378 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260927 18:34:20.662535 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.657109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:45611 (local address 127.5.151.254:41597)
0927 18:34:20.657148 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.657150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.657247 (+ 97us) server_negotiation.cc:407] Connection header received
0927 18:34:20.657318 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.657319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.657328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.657378 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.657687 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.658103 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.658810 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.659024 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.659095 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.660042 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.660043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.660043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.660049 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.661498 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.661791 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.661792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.661793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.661850 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.662005 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.662006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.662006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.662179 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.662239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.662337 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.662406 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:20.681746 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41597
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:20.760223 7191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:20.760399 7191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:20.760419 7191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:20.761452 7191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:20.761484 7191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:20.761504 7191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:20.761520 7191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:20.762913 7191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:41597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:20.763083 7191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:20.763237 7191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:20.767879 7199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:20.767956 7197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:20.767916 7191 server_base.cc:1034] running on GCE node
W20260927 18:34:20.767889 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:20.768324 7191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:20.768572 7191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:20.769702 7191 hybrid_clock.cc:648] HybridClock initialized: now 1790534060769691 us; error 25 us; skew 500 ppm
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:20.771981 7191 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:20.772387 7191 webserver.cc:460] Webserver started at http://127.5.151.195:42429/ using document root <none> and password file <none>
I20260927 18:34:20.772571 7191 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:20.772610 7191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:20.772722 7191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:20.773667 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "d037b9bc08514f11a88b2152b7c7fae5"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "a6dd2eb66de45dd83f4d7627019d45b5"
server_key_iv: "6140875cdc2fdc7064c1401551d000e1"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.773983 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "d037b9bc08514f11a88b2152b7c7fae5"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "a6dd2eb66de45dd83f4d7627019d45b5"
server_key_iv: "6140875cdc2fdc7064c1401551d000e1"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.775328 7191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:20.775959 7206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:20.776105 7191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:20.776175 7191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "d037b9bc08514f11a88b2152b7c7fae5"
format_stamp: "Formatted at 2026-09-27 18:34:20 on dist-test-slave-19nl"
server_key: "a6dd2eb66de45dd83f4d7627019d45b5"
server_key_iv: "6140875cdc2fdc7064c1401551d000e1"
server_key_version: "encryptionkey@0"
I20260927 18:34:20.776279 7191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:20.788363 7191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:20.789916 7191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:20.790014 7191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:20.790414 7191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:20.790448 7191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:20.790468 7191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:20.790508 7191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:20.797191 7191 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:37529
I20260927 18:34:20.797297 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:37529 every 8 connection(s)
I20260927 18:34:20.797560 7191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:20.800199 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.790811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:49891 (local address 127.5.151.254:41597)
0927 18:34:20.790871 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.790873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.791120 (+ 247us) server_negotiation.cc:407] Connection header received
0927 18:34:20.791789 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.791790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.791803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.791850 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.792725 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.793499 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.794914 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.795121 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.795199 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.797326 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.797328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.797329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.797336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.798863 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.799271 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.799273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.799273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.799331 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.799565 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.799566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.799567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.799673 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.799765 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.800018 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.800114 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:34:20.800329 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.790795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41597 (local address 127.5.151.195:49891)
0927 18:34:20.790982 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.791030 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.791666 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.791943 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.791947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.792328 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:20.792613 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.792620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.793577 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.793578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.794818 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.794821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.795219 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.795221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.795343 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.795816 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:20.795831 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:20.797243 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:20.798952 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.798958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.798960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.799201 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.799391 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.799392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.799392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.799493 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.799759 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:20.799764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:20.799888 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.799981 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.800132 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260927 18:34:20.800772 7322 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41597
I20260927 18:34:20.800861 7322 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:20.801012 7322 heartbeater.cc:511] Master 127.5.151.254:41597 requested a full tablet report, sending...
I20260927 18:34:20.801393 6839 ts_manager.cc:194] Registered new tserver with Master: d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195:37529)
I20260927 18:34:20.801782 6839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:49891
I20260927 18:34:20.805960 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 7191
I20260927 18:34:20.806034 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:20.806196 5727 external_mini_cluster.cc:1428] Setting key 8cf7049c47ce77f215675c0d2bb76f9f
I20260927 18:34:20.806908 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.801762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:52133 (local address 127.5.151.254:41597)
0927 18:34:20.801803 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.801804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.801902 (+ 98us) server_negotiation.cc:407] Connection header received
0927 18:34:20.801968 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.801969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.801977 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.802026 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.802316 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.802814 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.803517 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.803731 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.803787 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.804633 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.804634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.804634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.804641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.805730 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.806098 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.806099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.806099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.806156 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.806410 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.806411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.806411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.806500 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.806616 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.806764 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.806828 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:20.806926 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.801713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41597 (local address 127.5.151.195:52133)
0927 18:34:20.801852 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.801861 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.801914 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.802031 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.802032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.802201 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:20.802247 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.802248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.802894 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.802895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.803442 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.803443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.803785 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.803786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.803863 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.804276 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:20.804280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:20.804579 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:20.805792 (+ 1213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.805793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.805793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.805942 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.806216 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:20.806217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:20.806218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:20.806275 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:20.806610 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:20.806611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:20.806652 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.806739 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.806830 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260927 18:34:20.806936 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:34:20 dist-test-slave-19nl krb5kdc[6794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:20.827661 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.821629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.5.151.254:41597)
0927 18:34:20.821683 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.821685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.821715 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:34:20.821732 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.821732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.821740 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.821776 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:20.822280 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.822803 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.823452 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.823623 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.823670 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.825568 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:20.825569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:20.825569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:20.825575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:20.826859 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.827120 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.827121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.827121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.827167 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:20.827295 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:20.827295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:20.827295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:20.827373 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:20.827435 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.827505 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.827568 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:34:20.828969 6839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:20.830482 6839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:20.836374 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.832473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:39321 (local address 127.0.0.1:41500)
0927 18:34:20.832710 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.832748 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.832807 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.833050 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.833053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.833055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:20.833111 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.833112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.834200 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.834201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.835198 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.835200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.836095 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.836095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.836181 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.836195 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.836262 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.836307 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":64,"threads_started":1}
I20260927 18:34:20.836414 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.832699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.5.151.193:39321)
0927 18:34:20.832902 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.832907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.832915 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:20.832953 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.832955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.832967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.833045 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:20.833172 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.834136 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.835289 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.835999 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.836094 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.836149 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.836282 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.836321 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260927 18:34:20.837074 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.832486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:37227 (local address 127.0.0.1:59324)
0927 18:34:20.833628 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.833635 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.833684 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.833934 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.833934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.833936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:20.833966 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.833967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.835082 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.835084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.835904 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.835906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.836830 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.836831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.836893 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.836897 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.836953 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.836993 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":94,"threads_started":1}
I20260927 18:34:20.837091 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.833396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.5.151.194:37227)
0927 18:34:20.833792 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.833796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.833804 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:20.833838 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.833840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.833853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.833927 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:20.834018 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.835006 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.835976 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.836736 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.836826 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.836877 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.836967 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.837000 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":83,"threads_started":1}
I20260927 18:34:20.837028 6976 tablet_service.cc:1467] Processing CreateTablet for tablet e1991ad50c014328b1b46e20b8f9e3c0 (DEFAULT_TABLE table=test-table [id=c459b501f0b1446f831b910a4c58a4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:20.837393 6976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1991ad50c014328b1b46e20b8f9e3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:20.837649 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.832473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37529 (local address 127.0.0.1:47870)
0927 18:34:20.833318 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.833327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.833379 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.833585 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.833586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.833588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:20.833631 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.833632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.834822 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.834823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.835818 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.835820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.837071 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.837072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.837485 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.837492 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.837550 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.837587 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260927 18:34:20.837713 7116 tablet_service.cc:1467] Processing CreateTablet for tablet e1991ad50c014328b1b46e20b8f9e3c0 (DEFAULT_TABLE table=test-table [id=c459b501f0b1446f831b910a4c58a4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:20.837929 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.832685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.5.151.195:37529)
0927 18:34:20.833012 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.833017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.833439 (+ 422us) server_negotiation.cc:407] Connection header received
0927 18:34:20.833486 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.833488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.833502 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.833567 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:20.833700 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.834749 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.835903 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.836905 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.836971 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.837022 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.837799 (+ 777us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.837841 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":67,"threads_started":1}
I20260927 18:34:20.838058 7116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1991ad50c014328b1b46e20b8f9e3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:20.838495 7256 tablet_service.cc:1467] Processing CreateTablet for tablet e1991ad50c014328b1b46e20b8f9e3c0 (DEFAULT_TABLE table=test-table [id=c459b501f0b1446f831b910a4c58a4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:20.838843 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1991ad50c014328b1b46e20b8f9e3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:20.840951 7342 tablet_bootstrap.cc:492] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Bootstrap starting.
I20260927 18:34:20.841269 7343 tablet_bootstrap.cc:492] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Bootstrap starting.
I20260927 18:34:20.841591 7342 tablet_bootstrap.cc:654] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:20.841892 7343 tablet_bootstrap.cc:654] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:20.841928 7342 log.cc:824] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:20.842069 7344 tablet_bootstrap.cc:492] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Bootstrap starting.
I20260927 18:34:20.842233 7343 log.cc:824] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:20.842757 7342 tablet_bootstrap.cc:492] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: No bootstrap required, opened a new log
I20260927 18:34:20.842828 7342 ts_tablet_manager.cc:1397] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:20.842916 7343 tablet_bootstrap.cc:492] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: No bootstrap required, opened a new log
I20260927 18:34:20.842983 7344 tablet_bootstrap.cc:654] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:20.842993 7343 ts_tablet_manager.cc:1397] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:20.843300 7344 log.cc:824] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:20.843977 7344 tablet_bootstrap.cc:492] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: No bootstrap required, opened a new log
I20260927 18:34:20.844051 7344 ts_tablet_manager.cc:1397] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:20.844609 7343 raft_consensus.cc:348] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.844748 7343 raft_consensus.cc:374] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:20.844786 7343 raft_consensus.cc:729] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100c997f4b04333a9ea283aa6cf83c4, State: Initialized, Role: FOLLOWER
I20260927 18:34:20.844901 7343 consensus_queue.cc:260] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.845016 7342 raft_consensus.cc:348] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.845157 7042 heartbeater.cc:503] Master 127.5.151.254:41597 was elected leader, sending a full tablet report...
I20260927 18:34:20.845167 7342 raft_consensus.cc:374] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:20.845203 7342 raft_consensus.cc:729] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4fb3b01fc940f9b77efe7a47412b2a, State: Initialized, Role: FOLLOWER
I20260927 18:34:20.845304 7342 consensus_queue.cc:260] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.845396 7343 ts_tablet_manager.cc:1428] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:20.845552 7342 ts_tablet_manager.cc:1428] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:34:20.845675 7182 heartbeater.cc:503] Master 127.5.151.254:41597 was elected leader, sending a full tablet report...
I20260927 18:34:20.846316 7344 raft_consensus.cc:348] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.846486 7344 raft_consensus.cc:374] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:20.846532 7344 raft_consensus.cc:729] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d037b9bc08514f11a88b2152b7c7fae5, State: Initialized, Role: FOLLOWER
I20260927 18:34:20.846661 7344 consensus_queue.cc:260] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.846980 7344 ts_tablet_manager.cc:1428] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:34:20.847064 7322 heartbeater.cc:503] Master 127.5.151.254:41597 was elected leader, sending a full tablet report...
W20260927 18:34:20.904633 7183 tablet.cc:2369] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:20.967276 7349 raft_consensus.cc:482] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:20.967401 7349 raft_consensus.cc:504] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.967811 7349 leader_election.cc:288] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193:39321), d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195:37529)
I20260927 18:34:20.971832 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.968114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:33521 (local address 127.5.151.193:39321)
0927 18:34:20.968182 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.968185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.968199 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:34:20.968275 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.968276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.968287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.968357 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:20.968586 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.969314 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.970893 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.971460 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.971536 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.971553 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.971722 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.971758 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:34:20.971832 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.968151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:50651 (local address 127.5.151.195:37529)
0927 18:34:20.968198 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:20.968200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:20.968275 (+ 75us) server_negotiation.cc:407] Connection header received
0927 18:34:20.968313 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:20.968314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:20.968323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:20.968376 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:20.968571 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.969318 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.970891 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.971461 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.971536 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.971544 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:20.971722 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 18:34:20.971758 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 18:34:20.971864 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.968021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:39321 (local address 127.5.151.194:33521)
0927 18:34:20.968111 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.968125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.968193 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.968387 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.968387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.968389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:20.968501 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.968502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.969399 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.969400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.970804 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.970807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.971542 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.971543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.971645 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.971651 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.971708 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.971758 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":130}
I20260927 18:34:20.971872 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:20.968021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37529 (local address 127.5.151.194:50651)
0927 18:34:20.968205 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:20.968214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:20.968270 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:20.968387 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:20.968387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:20.968389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:20.968494 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.968496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.969399 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.969400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.970804 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:20.970807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:20.971542 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:20.971543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:20.971645 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:20.971652 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:34:20.971708 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:34:20.971755 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":65,"thread_start_us":84,"threads_started":1}
I20260927 18:34:20.972267 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1991ad50c014328b1b46e20b8f9e3c0" candidate_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100c997f4b04333a9ea283aa6cf83c4" is_pre_election: true
I20260927 18:34:20.972266 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1991ad50c014328b1b46e20b8f9e3c0" candidate_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d037b9bc08514f11a88b2152b7c7fae5" is_pre_election: true
I20260927 18:34:20.972419 6996 raft_consensus.cc:2454] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b4fb3b01fc940f9b77efe7a47412b2a in term 0.
I20260927 18:34:20.972424 7276 raft_consensus.cc:2454] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b4fb3b01fc940f9b77efe7a47412b2a in term 0.
I20260927 18:34:20.972671 7070 leader_election.cc:302] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b4fb3b01fc940f9b77efe7a47412b2a, d037b9bc08514f11a88b2152b7c7fae5; no voters:
I20260927 18:34:20.972801 7349 raft_consensus.cc:2781] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:20.972858 7349 raft_consensus.cc:482] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:20.972875 7349 raft_consensus.cc:3037] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:20.973839 7349 raft_consensus.cc:504] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.974017 7349 leader_election.cc:288] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [CANDIDATE]: Term 1 election: Requested vote from peers 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193:39321), d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195:37529)
I20260927 18:34:20.974162 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1991ad50c014328b1b46e20b8f9e3c0" candidate_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d037b9bc08514f11a88b2152b7c7fae5"
I20260927 18:34:20.974162 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1991ad50c014328b1b46e20b8f9e3c0" candidate_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100c997f4b04333a9ea283aa6cf83c4"
I20260927 18:34:20.974238 7276 raft_consensus.cc:3037] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:20.974247 6996 raft_consensus.cc:3037] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:20.975142 7276 raft_consensus.cc:2454] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b4fb3b01fc940f9b77efe7a47412b2a in term 1.
I20260927 18:34:20.975163 6996 raft_consensus.cc:2454] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b4fb3b01fc940f9b77efe7a47412b2a in term 1.
I20260927 18:34:20.975329 7067 leader_election.cc:302] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100c997f4b04333a9ea283aa6cf83c4, 9b4fb3b01fc940f9b77efe7a47412b2a; no voters:
I20260927 18:34:20.975449 7349 raft_consensus.cc:2781] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:20.975627 7349 raft_consensus.cc:686] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 LEADER]: Becoming Leader. State: Replica: 9b4fb3b01fc940f9b77efe7a47412b2a, State: Running, Role: LEADER
I20260927 18:34:20.975723 7349 consensus_queue.cc:237] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:20.976480 6839 catalog_manager.cc:5495] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194). New cstate: current_term: 1 leader_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } health_report { overall_health: UNKNOWN } } }
W20260927 18:34:20.997067 7043 tablet.cc:2369] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:21.016439 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.014149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.5.151.194:37227)
0927 18:34:21.014215 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.014216 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.014226 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:21.014334 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.014335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.014350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.014405 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:21.014519 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.015039 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.015755 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.015925 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.015972 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.016127 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.016211 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.016266 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.016308 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.016325 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:34:21.018126 7276 raft_consensus.cc:1261] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Refusing update from remote peer 9b4fb3b01fc940f9b77efe7a47412b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:21.018126 6996 raft_consensus.cc:1261] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Refusing update from remote peer 9b4fb3b01fc940f9b77efe7a47412b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:21.018486 7352 consensus_queue.cc:1035] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:21.018631 7349 consensus_queue.cc:1035] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:21.020792 7356 mvcc.cc:205] Tried to move back new op lower bound from 7334027513927409664 to 7334027513756925952. Current Snapshot: MvccSnapshot[applied={T|T < 7334027513927409664}]
I20260927 18:34:21.023921 6838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:21.024013 6838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:21.024755 6838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5: Sending DeleteTablet for 3 replicas of tablet e1991ad50c014328b1b46e20b8f9e3c0
I20260927 18:34:21.025059 6976 tablet_service.cc:1514] Processing DeleteTablet for tablet e1991ad50c014328b1b46e20b8f9e3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:41500
I20260927 18:34:21.025321 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet e1991ad50c014328b1b46e20b8f9e3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47870
I20260927 18:34:21.025373 7365 tablet_replica.cc:330] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: stopping tablet replica
I20260927 18:34:21.025388 7116 tablet_service.cc:1514] Processing DeleteTablet for tablet e1991ad50c014328b1b46e20b8f9e3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:59324
I20260927 18:34:21.025465 7365 raft_consensus.cc:2229] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:21.025561 7365 raft_consensus.cc:2258] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:21.025663 6838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:21.025841 6838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:21.026377 7366 tablet_replica.cc:330] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: stopping tablet replica
I20260927 18:34:21.026561 7366 raft_consensus.cc:2229] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:21.026648 7365 ts_tablet_manager.cc:1905] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:21.026763 7366 raft_consensus.cc:2258] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:21.027212 7366 ts_tablet_manager.cc:1905] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:21.027508 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 0407beb8b43040d989289c4bc81a48be (DEFAULT_TABLE table=test-table [id=bc19e80dac6a4675a68bb6710ac0d816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:21.027561 6976 tablet_service.cc:1467] Processing CreateTablet for tablet 0407beb8b43040d989289c4bc81a48be (DEFAULT_TABLE table=test-table [id=bc19e80dac6a4675a68bb6710ac0d816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:21.027617 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0407beb8b43040d989289c4bc81a48be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:21.027670 6976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0407beb8b43040d989289c4bc81a48be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:21.028424 7365 ts_tablet_manager.cc:1918] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:21.028483 7365 log.cc:1196] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/e1991ad50c014328b1b46e20b8f9e3c0
I20260927 18:34:21.028718 7365 ts_tablet_manager.cc:1939] T e1991ad50c014328b1b46e20b8f9e3c0 P 1100c997f4b04333a9ea283aa6cf83c4: Deleting consensus metadata
I20260927 18:34:21.029014 7343 tablet_bootstrap.cc:492] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: Bootstrap starting.
I20260927 18:34:21.029018 7344 tablet_bootstrap.cc:492] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Bootstrap starting.
I20260927 18:34:21.029170 7116 tablet_service.cc:1467] Processing CreateTablet for tablet 0407beb8b43040d989289c4bc81a48be (DEFAULT_TABLE table=test-table [id=bc19e80dac6a4675a68bb6710ac0d816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:21.029291 7116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0407beb8b43040d989289c4bc81a48be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:21.029390 7343 tablet_bootstrap.cc:654] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:21.029585 7344 tablet_bootstrap.cc:654] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:21.029851 6823 catalog_manager.cc:4841] TS 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193:39321): tablet e1991ad50c014328b1b46e20b8f9e3c0 (table test-table [id=c459b501f0b1446f831b910a4c58a4f7]) successfully deleted
I20260927 18:34:21.030424 7344 tablet_bootstrap.cc:492] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: No bootstrap required, opened a new log
I20260927 18:34:21.030479 7344 ts_tablet_manager.cc:1397] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:21.030648 7344 raft_consensus.cc:348] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.030704 7342 tablet_bootstrap.cc:492] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: Bootstrap starting.
I20260927 18:34:21.030745 7344 raft_consensus.cc:374] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:21.030772 7344 raft_consensus.cc:729] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d037b9bc08514f11a88b2152b7c7fae5, State: Initialized, Role: FOLLOWER
I20260927 18:34:21.030812 7344 consensus_queue.cc:260] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.030901 7344 ts_tablet_manager.cc:1428] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:21.031117 7342 tablet_bootstrap.cc:654] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:21.031374 7366 ts_tablet_manager.cc:1918] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:21.031502 7366 log.cc:1196] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/e1991ad50c014328b1b46e20b8f9e3c0
I20260927 18:34:21.031586 7343 tablet_bootstrap.cc:492] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: No bootstrap required, opened a new log
I20260927 18:34:21.031641 7343 ts_tablet_manager.cc:1397] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:34:21.031919 7366 ts_tablet_manager.cc:1939] T e1991ad50c014328b1b46e20b8f9e3c0 P d037b9bc08514f11a88b2152b7c7fae5: Deleting consensus metadata
I20260927 18:34:21.031984 7343 raft_consensus.cc:348] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.032079 7343 raft_consensus.cc:374] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:21.032105 7343 raft_consensus.cc:729] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100c997f4b04333a9ea283aa6cf83c4, State: Initialized, Role: FOLLOWER
I20260927 18:34:21.032155 7343 consensus_queue.cc:260] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.032263 7343 ts_tablet_manager.cc:1428] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:21.032500 7367 tablet_replica.cc:330] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: stopping tablet replica
I20260927 18:34:21.032532 6826 catalog_manager.cc:4841] TS d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195:37529): tablet e1991ad50c014328b1b46e20b8f9e3c0 (table test-table [id=c459b501f0b1446f831b910a4c58a4f7]) successfully deleted
I20260927 18:34:21.032683 7367 raft_consensus.cc:2229] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:21.032822 7367 raft_consensus.cc:2258] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:21.033203 7367 ts_tablet_manager.cc:1905] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:21.034382 7367 ts_tablet_manager.cc:1918] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:21.034473 7367 log.cc:1196] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/e1991ad50c014328b1b46e20b8f9e3c0
I20260927 18:34:21.034741 7367 ts_tablet_manager.cc:1939] T e1991ad50c014328b1b46e20b8f9e3c0 P 9b4fb3b01fc940f9b77efe7a47412b2a: Deleting consensus metadata
I20260927 18:34:21.035125 6824 catalog_manager.cc:4841] TS 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194:37227): tablet e1991ad50c014328b1b46e20b8f9e3c0 (table test-table [id=c459b501f0b1446f831b910a4c58a4f7]) successfully deleted
I20260927 18:34:21.040372 7342 tablet_bootstrap.cc:492] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: No bootstrap required, opened a new log
I20260927 18:34:21.040432 7342 ts_tablet_manager.cc:1397] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260927 18:34:21.040582 7342 raft_consensus.cc:348] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.040655 7342 raft_consensus.cc:374] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:21.040686 7342 raft_consensus.cc:729] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4fb3b01fc940f9b77efe7a47412b2a, State: Initialized, Role: FOLLOWER
I20260927 18:34:21.040733 7342 consensus_queue.cc:260] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.040813 7342 ts_tablet_manager.cc:1428] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260927 18:34:21.048425 7323 tablet.cc:2369] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:21.438227 7348 raft_consensus.cc:482] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:21.438338 7348 raft_consensus.cc:504] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.438759 7348 leader_election.cc:288] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194:37227), d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195:37529)
I20260927 18:34:21.441679 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.438996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:42333 (local address 127.5.151.195:37529)
0927 18:34:21.439094 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.439096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.439147 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:34:21.439216 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.439217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.439226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.439276 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:21.439417 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.439974 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.440970 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.441321 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.441399 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.441409 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.441572 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.441603 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:34:21.441694 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.439147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:60799 (local address 127.5.151.194:37227)
0927 18:34:21.439192 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.439194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.439201 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:21.439225 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.439226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.439239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.439307 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:21.439457 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.439968 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.440972 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.441328 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.441403 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.441412 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.441582 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.441626 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 18:34:21.441738 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.438944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37529 (local address 127.5.151.193:42333)
0927 18:34:21.439076 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.439090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.439160 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.439278 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.439278 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.439280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:21.439349 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.439351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.440069 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.440070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.440885 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.440887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.441428 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.441430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.441495 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.441499 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.441564 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.441602 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":31,"thread_start_us":89,"threads_started":1}
I20260927 18:34:21.441972 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0407beb8b43040d989289c4bc81a48be" candidate_uuid: "1100c997f4b04333a9ea283aa6cf83c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d037b9bc08514f11a88b2152b7c7fae5" is_pre_election: true
I20260927 18:34:21.442051 7276 raft_consensus.cc:2454] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100c997f4b04333a9ea283aa6cf83c4 in term 0.
I20260927 18:34:21.442143 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.438921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:37227 (local address 127.5.151.193:60799)
0927 18:34:21.439032 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.439045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.439113 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.439313 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.439314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.439315 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:21.439385 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.439387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.440050 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.440052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.440884 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.440886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.441430 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.441431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.441504 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.441507 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.441574 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.441616 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":29,"thread_start_us":65,"threads_started":1}
I20260927 18:34:21.442220 6930 leader_election.cc:302] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100c997f4b04333a9ea283aa6cf83c4, d037b9bc08514f11a88b2152b7c7fae5; no voters:
I20260927 18:34:21.442332 7348 raft_consensus.cc:2781] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:21.442366 7348 raft_consensus.cc:482] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:21.442384 7348 raft_consensus.cc:3037] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:21.442440 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0407beb8b43040d989289c4bc81a48be" candidate_uuid: "1100c997f4b04333a9ea283aa6cf83c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" is_pre_election: true
I20260927 18:34:21.442533 7136 raft_consensus.cc:2454] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100c997f4b04333a9ea283aa6cf83c4 in term 0.
I20260927 18:34:21.443104 7348 raft_consensus.cc:504] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.443246 7348 leader_election.cc:288] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [CANDIDATE]: Term 1 election: Requested vote from peers 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194:37227), d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195:37529)
I20260927 18:34:21.443378 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0407beb8b43040d989289c4bc81a48be" candidate_uuid: "1100c997f4b04333a9ea283aa6cf83c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a"
I20260927 18:34:21.443397 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0407beb8b43040d989289c4bc81a48be" candidate_uuid: "1100c997f4b04333a9ea283aa6cf83c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d037b9bc08514f11a88b2152b7c7fae5"
I20260927 18:34:21.443468 7276 raft_consensus.cc:3037] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:21.443470 7136 raft_consensus.cc:3037] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:21.444015 7276 raft_consensus.cc:2454] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100c997f4b04333a9ea283aa6cf83c4 in term 1.
I20260927 18:34:21.444159 6930 leader_election.cc:302] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100c997f4b04333a9ea283aa6cf83c4, d037b9bc08514f11a88b2152b7c7fae5; no voters:
I20260927 18:34:21.444162 7136 raft_consensus.cc:2454] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100c997f4b04333a9ea283aa6cf83c4 in term 1.
I20260927 18:34:21.444243 7348 raft_consensus.cc:2781] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:21.444376 7348 raft_consensus.cc:686] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 LEADER]: Becoming Leader. State: Replica: 1100c997f4b04333a9ea283aa6cf83c4, State: Running, Role: LEADER
I20260927 18:34:21.444456 7348 consensus_queue.cc:237] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.445076 6838 catalog_manager.cc:5495] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "1100c997f4b04333a9ea283aa6cf83c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:34:21 dist-test-slave-19nl krb5kdc[6794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:21.493587 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.487273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.5.151.254:41597)
0927 18:34:21.487431 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.487433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.487461 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:34:21.487489 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.487490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.487500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.487592 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:21.488019 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.488521 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.489143 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.489320 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.489369 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.491296 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:21.491297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:21.491298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:21.491303 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:21.492652 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:21.492981 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:21.492982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:21.492982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:21.493034 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:21.493194 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:21.493194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:21.493195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:21.493292 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:21.493342 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.493451 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.493517 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260927 18:34:21.493754 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.487187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41597 (local address 127.0.0.1:49150)
0927 18:34:21.487330 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.487342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.487417 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.487599 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.487599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.487903 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:21.487949 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.487951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.488581 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.488581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.489079 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.489080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.489377 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.489378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.489416 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.489865 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:21.489869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:21.491229 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:21.492719 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:21.492724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:21.492727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:21.492904 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:21.493087 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:21.493088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:21.493088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:21.493146 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:21.493340 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:21.493346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:21.493386 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.493435 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.493581 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":60,"thread_start_us":65,"threads_started":1}
I20260927 18:34:21.495609 6838 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:49150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:34:21.496095 6838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:21.497467 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 5d93c7c3a7b140b58dca72894ff6168c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b348664f969a48829f13b4eb02dec0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:21.497473 7116 tablet_service.cc:1467] Processing CreateTablet for tablet 5d93c7c3a7b140b58dca72894ff6168c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b348664f969a48829f13b4eb02dec0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:21.497504 6976 tablet_service.cc:1467] Processing CreateTablet for tablet 5d93c7c3a7b140b58dca72894ff6168c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b348664f969a48829f13b4eb02dec0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:21.497627 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d93c7c3a7b140b58dca72894ff6168c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:21.497628 7116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d93c7c3a7b140b58dca72894ff6168c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:21.497648 6976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d93c7c3a7b140b58dca72894ff6168c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:21.499137 7342 tablet_bootstrap.cc:492] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: Bootstrap starting.
I20260927 18:34:21.499159 7344 tablet_bootstrap.cc:492] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: Bootstrap starting.
I20260927 18:34:21.499315 7343 tablet_bootstrap.cc:492] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: Bootstrap starting.
I20260927 18:34:21.499501 7342 tablet_bootstrap.cc:654] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:21.499583 7343 tablet_bootstrap.cc:654] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:21.499593 7344 tablet_bootstrap.cc:654] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:21.500104 7342 tablet_bootstrap.cc:492] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: No bootstrap required, opened a new log
I20260927 18:34:21.500186 7342 ts_tablet_manager.cc:1397] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:21.500366 7344 tablet_bootstrap.cc:492] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: No bootstrap required, opened a new log
I20260927 18:34:21.500305 7342 raft_consensus.cc:348] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.500415 7342 raft_consensus.cc:374] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:21.500437 7342 raft_consensus.cc:729] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4fb3b01fc940f9b77efe7a47412b2a, State: Initialized, Role: FOLLOWER
I20260927 18:34:21.500428 7344 ts_tablet_manager.cc:1397] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:21.500488 7342 consensus_queue.cc:260] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.500593 7342 ts_tablet_manager.cc:1428] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:21.500568 7344 raft_consensus.cc:348] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.500633 7344 raft_consensus.cc:374] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:21.500654 7344 raft_consensus.cc:729] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d037b9bc08514f11a88b2152b7c7fae5, State: Initialized, Role: FOLLOWER
I20260927 18:34:21.500696 7344 consensus_queue.cc:260] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.500660 7362 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.500747 7362 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:21.500778 7344 ts_tablet_manager.cc:1428] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:21.500979 7343 tablet_bootstrap.cc:492] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: No bootstrap required, opened a new log
I20260927 18:34:21.501029 7343 ts_tablet_manager.cc:1397] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:21.501000 7350 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.501118 7350 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:21.501176 7343 raft_consensus.cc:348] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.501246 7343 raft_consensus.cc:374] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:21.501260 7343 raft_consensus.cc:729] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100c997f4b04333a9ea283aa6cf83c4, State: Initialized, Role: FOLLOWER
I20260927 18:34:21.501319 7343 consensus_queue.cc:260] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.501422 7343 ts_tablet_manager.cc:1428] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:21.501477 7348 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.501593 7348 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:21.620349 7350 raft_consensus.cc:482] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:21.620457 7350 raft_consensus.cc:504] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.620831 7350 leader_election.cc:288] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193:39321), 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194:37227)
I20260927 18:34:21.623814 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.621008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:39321 (local address 127.5.151.195:34511)
0927 18:34:21.621158 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.621171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.621231 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.621420 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.621421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.621423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:21.621480 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.621482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.622149 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.622150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.622982 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.622984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.623547 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.623548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.623618 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.623621 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.623681 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.623723 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260927 18:34:21.623804 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.621090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:34511 (local address 127.5.151.193:39321)
0927 18:34:21.621225 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.621227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.621238 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:34:21.621296 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.621297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.621312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.621409 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:21.621560 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.622063 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.623079 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.623455 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.623535 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.623553 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.623699 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.623737 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":94,"threads_started":1}
I20260927 18:34:21.624058 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.621108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:40639 (local address 127.5.151.194:37227)
0927 18:34:21.621161 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.621163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.621357 (+ 194us) server_negotiation.cc:407] Connection header received
0927 18:34:21.621435 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.621436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.621446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.621503 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:21.621664 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.622302 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.623349 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.623717 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.623807 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.623815 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.623952 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.624000 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:34:21.624078 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.621031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:37227 (local address 127.5.151.195:40639)
0927 18:34:21.621278 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.621290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.621372 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.621507 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.621508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.621509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:21.621573 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.621575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.622376 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.622377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.623265 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.623267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.623809 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.623810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.623879 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.623883 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.623942 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.623990 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":62,"threads_started":1}
I20260927 18:34:21.624296 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93c7c3a7b140b58dca72894ff6168c" candidate_uuid: "d037b9bc08514f11a88b2152b7c7fae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100c997f4b04333a9ea283aa6cf83c4" is_pre_election: true
I20260927 18:34:21.624416 6996 raft_consensus.cc:2454] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d037b9bc08514f11a88b2152b7c7fae5 in term 0.
I20260927 18:34:21.624369 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93c7c3a7b140b58dca72894ff6168c" candidate_uuid: "d037b9bc08514f11a88b2152b7c7fae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" is_pre_election: true
I20260927 18:34:21.624469 7136 raft_consensus.cc:2454] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d037b9bc08514f11a88b2152b7c7fae5 in term 0.
I20260927 18:34:21.624606 7207 leader_election.cc:302] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100c997f4b04333a9ea283aa6cf83c4, d037b9bc08514f11a88b2152b7c7fae5; no voters:
I20260927 18:34:21.624750 7350 raft_consensus.cc:2781] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:21.624799 7350 raft_consensus.cc:482] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:21.624821 7350 raft_consensus.cc:3037] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:21.625507 7350 raft_consensus.cc:504] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.625649 7350 leader_election.cc:288] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [CANDIDATE]: Term 1 election: Requested vote from peers 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193:39321), 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194:37227)
I20260927 18:34:21.625794 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93c7c3a7b140b58dca72894ff6168c" candidate_uuid: "d037b9bc08514f11a88b2152b7c7fae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1100c997f4b04333a9ea283aa6cf83c4"
I20260927 18:34:21.625847 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93c7c3a7b140b58dca72894ff6168c" candidate_uuid: "d037b9bc08514f11a88b2152b7c7fae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a"
I20260927 18:34:21.625891 6996 raft_consensus.cc:3037] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:21.625902 7136 raft_consensus.cc:3037] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:21.626511 7136 raft_consensus.cc:2454] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d037b9bc08514f11a88b2152b7c7fae5 in term 1.
I20260927 18:34:21.626518 6996 raft_consensus.cc:2454] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d037b9bc08514f11a88b2152b7c7fae5 in term 1.
I20260927 18:34:21.626704 7208 leader_election.cc:302] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b4fb3b01fc940f9b77efe7a47412b2a, d037b9bc08514f11a88b2152b7c7fae5; no voters:
I20260927 18:34:21.626806 7350 raft_consensus.cc:2781] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:21.626922 7350 raft_consensus.cc:686] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [term 1 LEADER]: Becoming Leader. State: Replica: d037b9bc08514f11a88b2152b7c7fae5, State: Running, Role: LEADER
I20260927 18:34:21.626992 7350 consensus_queue.cc:237] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } }
I20260927 18:34:21.627295 7388 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: TxnStatusTablet state changed. Reason: New leader d037b9bc08514f11a88b2152b7c7fae5. Latest consensus state: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.627377 7388 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:21.627542 7390 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:34:21.627656 7390 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:34:21.627713 6839 catalog_manager.cc:5495] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } health_report { overall_health: HEALTHY } } }
I20260927 18:34:21.649204 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.646633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37529 (local address 127.0.0.1:47880)
0927 18:34:21.646773 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.646783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.646842 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.647032 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.647033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.647034 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:21.647086 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.647089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.647720 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.647721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.648308 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.648310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.648690 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.648691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.648759 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.648778 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.649029 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.649039 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.649078 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.649134 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 18:34:21.649202 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.646701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.5.151.195:37529)
0927 18:34:21.646747 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.646750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.646822 (+ 72us) server_negotiation.cc:407] Connection header received
0927 18:34:21.646917 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.646918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.646939 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.647024 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:21.647154 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.647641 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.648413 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.648605 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.648688 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.648868 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.648965 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.649047 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.649094 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.649112 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:21.650686 6996 raft_consensus.cc:1261] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Refusing update from remote peer d037b9bc08514f11a88b2152b7c7fae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:21.650799 7136 raft_consensus.cc:1261] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Refusing update from remote peer d037b9bc08514f11a88b2152b7c7fae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:21.650960 7388 consensus_queue.cc:1035] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:21.651091 7350 consensus_queue.cc:1035] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:21.651737 7348 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: TxnStatusTablet state changed. Reason: New leader d037b9bc08514f11a88b2152b7c7fae5. Latest consensus state: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.651835 7348 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:21.651808 7362 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: TxnStatusTablet state changed. Reason: New leader d037b9bc08514f11a88b2152b7c7fae5. Latest consensus state: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.651894 7362 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:21.652679 7350 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.652766 7350 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:21.652781 7394 mvcc.cc:205] Tried to move back new op lower bound from 7334027516518670336 to 7334027516424482816. Current Snapshot: MvccSnapshot[applied={T|T < 7334027516518670336}]
I20260927 18:34:21.652884 7362 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.652964 7362 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P 9b4fb3b01fc940f9b77efe7a47412b2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:21.653283 7350 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.653391 7350 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P d037b9bc08514f11a88b2152b7c7fae5: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:21.653576 7393 mvcc.cc:205] Tried to move back new op lower bound from 7334027516518670336 to 7334027516424482816. Current Snapshot: MvccSnapshot[applied={T|T < 7334027516518670336}]
I20260927 18:34:21.653693 7348 tablet_replica.cc:439] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d037b9bc08514f11a88b2152b7c7fae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100c997f4b04333a9ea283aa6cf83c4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 39321 } } peers { permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 } } peers { permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 } } }
I20260927 18:34:21.653771 7348 tablet_replica.cc:442] T 5d93c7c3a7b140b58dca72894ff6168c P 1100c997f4b04333a9ea283aa6cf83c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:21.659970 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.655616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.5.151.193:39321)
0927 18:34:21.655662 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.655664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.657485 (+ 1821us) server_negotiation.cc:407] Connection header received
0927 18:34:21.657573 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.657573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.657593 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.657656 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:21.657849 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.658326 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.659148 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.659350 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.659453 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.659627 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.659733 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.659825 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.659866 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.659886 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:34:21.664597 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.662332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:45405 (local address 127.5.151.195:37529)
0927 18:34:21.662379 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.662381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.662459 (+ 78us) server_negotiation.cc:407] Connection header received
0927 18:34:21.662528 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.662529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.662537 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.662629 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:21.662755 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.663188 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.663919 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.664110 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.664189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.664358 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.664390 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.664454 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.664499 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.664516 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 18:34:21.664623 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.662268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37529 (local address 127.5.151.193:45405)
0927 18:34:21.662404 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.662414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.662471 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.662633 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.662633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.662635 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:21.662696 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.662698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.663258 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.663260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.663833 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.663835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.664190 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.664191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.664254 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.664268 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.664444 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.664446 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.664486 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.664528 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260927 18:34:21.667155 7276 raft_consensus.cc:1261] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Refusing update from remote peer 1100c997f4b04333a9ea283aa6cf83c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:21.667155 7136 raft_consensus.cc:1261] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Refusing update from remote peer 1100c997f4b04333a9ea283aa6cf83c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:21.667397 7373 consensus_queue.cc:1035] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b4fb3b01fc940f9b77efe7a47412b2a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:21.667524 7348 consensus_queue.cc:1035] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d037b9bc08514f11a88b2152b7c7fae5" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:21.676283 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.673483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:41761 (local address 127.5.151.193:39321)
0927 18:34:21.673530 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.673532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.673657 (+ 125us) server_negotiation.cc:407] Connection header received
0927 18:34:21.673751 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.673751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.673762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.673813 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:21.673966 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.674417 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.675629 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.675807 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.675857 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.676047 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.676075 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.676107 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.676162 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.676178 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:21.676297 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.673431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:39321 (local address 127.5.151.195:41761)
0927 18:34:21.673602 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:21.673613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:21.673671 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:21.673826 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:21.673827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:21.673829 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:21.673901 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.673902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.674484 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.674485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.675505 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:21.675508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.675871 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.675872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:21.675942 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.675958 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:21.676110 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:21.676111 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 18:34:21.676160 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:34:21.676201 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260927 18:34:21.685534 6838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:21.685664 6838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:21.686262 6838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad72486fd8464142b7548f36dbb5d6c5: Sending DeleteTablet for 3 replicas of tablet 0407beb8b43040d989289c4bc81a48be
I20260927 18:34:21.686522 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet 0407beb8b43040d989289c4bc81a48be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47870
I20260927 18:34:21.686726 7116 tablet_service.cc:1514] Processing DeleteTablet for tablet 0407beb8b43040d989289c4bc81a48be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:59324
I20260927 18:34:21.686935 6976 tablet_service.cc:1514] Processing DeleteTablet for tablet 0407beb8b43040d989289c4bc81a48be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:41500
I20260927 18:34:21.687264 7417 tablet_replica.cc:330] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: stopping tablet replica
I20260927 18:34:21.687361 7417 raft_consensus.cc:2229] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:21.687493 7417 raft_consensus.cc:2258] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:21.687800 7417 ts_tablet_manager.cc:1905] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:21.689114 7416 tablet_replica.cc:330] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: stopping tablet replica
I20260927 18:34:21.689198 7416 raft_consensus.cc:2229] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:21.689272 7416 raft_consensus.cc:2258] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:21.689396 7417 ts_tablet_manager.cc:1918] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:21.689493 7417 log.cc:1196] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/0407beb8b43040d989289c4bc81a48be
I20260927 18:34:21.689540 7416 ts_tablet_manager.cc:1905] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:21.689844 7417 ts_tablet_manager.cc:1939] T 0407beb8b43040d989289c4bc81a48be P 1100c997f4b04333a9ea283aa6cf83c4: Deleting consensus metadata
I20260927 18:34:21.690173 6823 catalog_manager.cc:4841] TS 1100c997f4b04333a9ea283aa6cf83c4 (127.5.151.193:39321): tablet 0407beb8b43040d989289c4bc81a48be (table test-table [id=bc19e80dac6a4675a68bb6710ac0d816]) successfully deleted
I20260927 18:34:21.691000 7416 ts_tablet_manager.cc:1918] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:21.691079 7416 log.cc:1196] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/0407beb8b43040d989289c4bc81a48be
I20260927 18:34:21.691285 7416 ts_tablet_manager.cc:1939] T 0407beb8b43040d989289c4bc81a48be P 9b4fb3b01fc940f9b77efe7a47412b2a: Deleting consensus metadata
I20260927 18:34:21.691596 7415 tablet_replica.cc:330] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: stopping tablet replica
I20260927 18:34:21.691639 6824 catalog_manager.cc:4841] TS 9b4fb3b01fc940f9b77efe7a47412b2a (127.5.151.194:37227): tablet 0407beb8b43040d989289c4bc81a48be (table test-table [id=bc19e80dac6a4675a68bb6710ac0d816]) successfully deleted
I20260927 18:34:21.691658 7415 raft_consensus.cc:2229] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:21.691696 7415 raft_consensus.cc:2258] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:21.691926 7415 ts_tablet_manager.cc:1905] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:21.693043 7415 ts_tablet_manager.cc:1918] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:21.693126 7415 log.cc:1196] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/0407beb8b43040d989289c4bc81a48be
I20260927 18:34:21.693326 7415 ts_tablet_manager.cc:1939] T 0407beb8b43040d989289c4bc81a48be P d037b9bc08514f11a88b2152b7c7fae5: Deleting consensus metadata
I20260927 18:34:21.693653 6826 catalog_manager.cc:4841] TS d037b9bc08514f11a88b2152b7c7fae5 (127.5.151.195:37529): tablet 0407beb8b43040d989289c4bc81a48be (table test-table [id=bc19e80dac6a4675a68bb6710ac0d816]) successfully deleted
Sep 27 18:34:21 dist-test-slave-19nl krb5kdc[6794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:21.701846 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.687425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.5.151.193:39321)
0927 18:34:21.687461 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.687464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.693686 (+ 6222us) server_negotiation.cc:407] Connection header received
0927 18:34:21.693802 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.693803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.693813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.693863 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:21.694397 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.694939 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.695640 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.695838 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.695917 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.698170 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:21.698179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:21.698180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:21.698187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:21.700040 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:21.700461 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:21.700464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:21.700465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:21.700528 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:21.701260 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:21.701260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:21.701261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:21.701465 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:21.701625 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.701693 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.701773 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 18:34:21.702365 7036 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41512
I20260927 18:34:21.712939 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:21.703342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.5.151.254:41597)
0927 18:34:21.703391 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:21.703393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:21.706715 (+ 3322us) server_negotiation.cc:407] Connection header received
0927 18:34:21.706793 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:21.706794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:21.706827 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:21.706897 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:21.707398 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.707920 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.708722 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:21.708927 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:21.708997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:21.710245 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:21.710246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:21.710246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:21.710253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:21.711740 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:21.712116 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:21.712118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:21.712118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:21.712182 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:21.712405 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:21.712406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:21.712407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:21.712527 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:21.712638 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:21.712746 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 18:34:21.712831 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 18:34:21.713284 6838 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49164
I20260927 18:34:21.713934 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 6911
I20260927 18:34:21.720546 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 7051
I20260927 18:34:21.727828 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 7191
I20260927 18:34:21.732932 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 6807
2026-09-27T18:34:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2678 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:21 dist-test-slave-19nl krb5kdc[7424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:21 dist-test-slave-19nl krb5kdc[7424](info): set up 2 sockets
Sep 27 18:34:21 dist-test-slave-19nl krb5kdc[7424](info): commencing operation
krb5kdc: starting...
W20260927 18:34:22.766217 5727 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:22 dist-test-slave-19nl krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:22.880632 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34949
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:34949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:22.959522 7437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:22.959753 7437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:22.959800 7437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:22.960947 7437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:22.961009 7437 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:34:22.961030 7437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:22.961061 7437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:22.961089 7437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:22.961112 7437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:22.962550 7437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:34949
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:22.962823 7437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:22.963037 7437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:22.967736 7437 server_base.cc:1034] running on GCE node
W20260927 18:34:22.967743 7443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:22.967818 7445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:22.968004 7442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:22.968284 7437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:22.968534 7437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:22.969677 7437 hybrid_clock.cc:648] HybridClock initialized: now 1790534062969661 us; error 29 us; skew 500 ppm
Sep 27 18:34:22 dist-test-slave-19nl krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:22.972128 7437 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:22.972579 7437 webserver.cc:460] Webserver started at http://127.5.151.254:34203/ using document root <none> and password file <none>
I20260927 18:34:22.972795 7437 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:22.972856 7437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:22.972997 7437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:22.974032 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "b2617ac70abc4cb1b940efde4263189f"
format_stamp: "Formatted at 2026-09-27 18:34:22 on dist-test-slave-19nl"
server_key: "8941e1852eb367b96d8a6eadb344fdfc"
server_key_iv: "56c1927118f7d5d292c45b83448386f5"
server_key_version: "encryptionkey@0"
I20260927 18:34:22.974387 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "b2617ac70abc4cb1b940efde4263189f"
format_stamp: "Formatted at 2026-09-27 18:34:22 on dist-test-slave-19nl"
server_key: "8941e1852eb367b96d8a6eadb344fdfc"
server_key_iv: "56c1927118f7d5d292c45b83448386f5"
server_key_version: "encryptionkey@0"
I20260927 18:34:22.975760 7437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:22.976478 7452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:22.976634 7437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:22.976737 7437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "b2617ac70abc4cb1b940efde4263189f"
format_stamp: "Formatted at 2026-09-27 18:34:22 on dist-test-slave-19nl"
server_key: "8941e1852eb367b96d8a6eadb344fdfc"
server_key_iv: "56c1927118f7d5d292c45b83448386f5"
server_key_version: "encryptionkey@0"
I20260927 18:34:22.976791 7437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:23.002262 7437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:23.004112 7437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:23.004222 7437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:23.008797 7437 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:34949
I20260927 18:34:23.008847 7514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:34949 every 8 connection(s)
I20260927 18:34:23.009289 7437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:23.009999 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.012274 7515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Bootstrap starting.
I20260927 18:34:23.012851 7515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.013151 7515 log.cc:824] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:23.013855 7515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: No bootstrap required, opened a new log
I20260927 18:34:23.015043 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 7437
I20260927 18:34:23.015136 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:23.015316 5727 external_mini_cluster.cc:1428] Setting key a36bcbaf04994d9347a04487996ed7d6
I20260927 18:34:23.016069 7515 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2617ac70abc4cb1b940efde4263189f" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34949 } }
I20260927 18:34:23.016253 7515 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.016295 7515 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2617ac70abc4cb1b940efde4263189f, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.016427 7515 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2617ac70abc4cb1b940efde4263189f" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34949 } }
I20260927 18:34:23.016506 7515 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:23.016556 7515 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:23.016603 7515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:23.017319 7515 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2617ac70abc4cb1b940efde4263189f" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34949 } }
I20260927 18:34:23.017443 7515 leader_election.cc:302] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2617ac70abc4cb1b940efde4263189f; no voters:
I20260927 18:34:23.017625 7515 leader_election.cc:288] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:23.017654 7520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:23.017769 7520 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [term 1 LEADER]: Becoming Leader. State: Replica: b2617ac70abc4cb1b940efde4263189f, State: Running, Role: LEADER
I20260927 18:34:23.017933 7515 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:23.017918 7520 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2617ac70abc4cb1b940efde4263189f" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34949 } }
I20260927 18:34:23.018399 7521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2617ac70abc4cb1b940efde4263189f. Latest consensus state: current_term: 1 leader_uuid: "b2617ac70abc4cb1b940efde4263189f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2617ac70abc4cb1b940efde4263189f" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34949 } } }
I20260927 18:34:23.018493 7521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:23.018749 7522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2617ac70abc4cb1b940efde4263189f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2617ac70abc4cb1b940efde4263189f" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34949 } } }
I20260927 18:34:23.018864 7522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:23.019043 7528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:23.019613 7528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:23.021229 7528 catalog_manager.cc:1347] Generated new cluster ID: d48e0dcc519c43179ba13da5fb351f0c
I20260927 18:34:23.021284 7528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:23.026998 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.016035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.5.151.254:34949)
0927 18:34:23.016218 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.016227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.016261 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:34:23.016385 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.016392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.016448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.016593 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.017534 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.018840 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.019940 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.020546 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.020650 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.023478 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.023482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.023483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.023494 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.025387 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.025778 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.025781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.025781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.025901 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.026108 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.026109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.026109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.026381 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.026522 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.026677 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.026809 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260927 18:34:23.056857 7528 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:23.057425 7528 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:23.064719 7528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Generated new TSK 0
I20260927 18:34:23.064930 7528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:23.095484 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34949
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:23.175103 7541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:23.175295 7541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:23.175319 7541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:23.176393 7541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:23.176427 7541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:23.176446 7541 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:34:23.176457 7541 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:23.176474 7541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:23.177842 7541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:23.178009 7541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:23.178186 7541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:23.183158 7541 server_base.cc:1034] running on GCE node
W20260927 18:34:23.183120 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:23.183198 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:23.183125 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:23.183506 7541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:23.183696 7541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:23.184854 7541 hybrid_clock.cc:648] HybridClock initialized: now 1790534063184842 us; error 42 us; skew 500 ppm
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:23.187162 7541 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:23.187557 7541 webserver.cc:460] Webserver started at http://127.5.151.193:42619/ using document root <none> and password file <none>
I20260927 18:34:23.187778 7541 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:23.187852 7541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:23.187989 7541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:23.188977 7541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "cb6e19f88b404f32a41a8b9bb3e43780"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "4a2182997409b716f183d3d13ef00426"
server_key_iv: "561809cf991b70dd81241334bc0ade88"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.189337 7541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "cb6e19f88b404f32a41a8b9bb3e43780"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "4a2182997409b716f183d3d13ef00426"
server_key_iv: "561809cf991b70dd81241334bc0ade88"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.190666 7541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 18:34:23.191391 7556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.191547 7541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:23.191638 7541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "cb6e19f88b404f32a41a8b9bb3e43780"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "4a2182997409b716f183d3d13ef00426"
server_key_iv: "561809cf991b70dd81241334bc0ade88"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.191715 7541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:23.205780 7541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:23.207559 7541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:23.207715 7541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:23.208191 7541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:23.208253 7541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.208297 7541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:23.208321 7541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:23.215489 7541 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:42027
I20260927 18:34:23.215682 7671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:42027 every 8 connection(s)
I20260927 18:34:23.215889 7541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:23.217878 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.208721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:43085 (local address 127.5.151.254:34949)
0927 18:34:23.208801 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.208804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.208960 (+ 156us) server_negotiation.cc:407] Connection header received
0927 18:34:23.209435 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.209437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.209450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.209521 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.210374 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.211231 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.212304 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.212509 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.212591 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.215091 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.215093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.215093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.215099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.216790 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.217129 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.217130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.217130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.217182 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.217360 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.217361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.217361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.217457 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.217504 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.217703 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.217801 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:34:23.218094 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.208675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34949 (local address 127.5.151.193:43085)
0927 18:34:23.208829 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.208870 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.209335 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.209644 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.209648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.209980 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.210259 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.210265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.211363 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.211365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.212212 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.212215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.212614 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.212615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.212739 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.213200 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.213210 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.214992 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.216867 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.216871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.216872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.217072 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.217225 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.217225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.217225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.217312 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.217507 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.217511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.217613 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.217692 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.217852 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260927 18:34:23.219385 7672 heartbeater.cc:347] Connected to a master server at 127.5.151.254:34949
I20260927 18:34:23.219446 7672 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:23.219611 7672 heartbeater.cc:511] Master 127.5.151.254:34949 requested a full tablet report, sending...
I20260927 18:34:23.219800 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 7541
I20260927 18:34:23.219899 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:23.220074 5727 external_mini_cluster.cc:1428] Setting key 600ba8b35e239d3cdba9f9fb14da2e0c
I20260927 18:34:23.220213 7469 ts_manager.cc:194] Registered new tserver with Master: cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027)
I20260927 18:34:23.221074 7469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:43085
I20260927 18:34:23.225445 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.219482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34949 (local address 127.5.151.193:55567)
0927 18:34:23.219609 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.219621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.219724 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.219909 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.219910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.220097 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.220150 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.220152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.221035 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.221036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.221691 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.221693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.222118 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.222119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.222183 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.222592 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.222596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.222851 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.224564 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.224565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.224565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.224722 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.224915 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.224916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.224916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.224964 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.225223 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.225224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.225254 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.225310 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.225378 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260927 18:34:23.225549 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.219662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:55567 (local address 127.5.151.254:34949)
0927 18:34:23.219701 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.219703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.219711 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:23.219796 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.219797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.219806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.219844 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.220234 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.220945 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.221776 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.222030 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.222121 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.222928 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.222929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.222930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.222937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.224489 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.224791 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.224792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.224793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.224852 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.225019 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.225020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.225021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.225161 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.225237 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.225344 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.225423 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:23.243013 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34949
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:23.321986 7681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:23.322196 7681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:23.322235 7681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:23.323380 7681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:23.323436 7681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:23.323468 7681 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:34:23.323496 7681 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:23.323529 7681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:23.324889 7681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:23.325132 7681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:23.325347 7681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:23.330025 7686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:23.330093 7687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:23.330091 7689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:23.330148 7681 server_base.cc:1034] running on GCE node
I20260927 18:34:23.330475 7681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:23.330715 7681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:23.331871 7681 hybrid_clock.cc:648] HybridClock initialized: now 1790534063331857 us; error 28 us; skew 500 ppm
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:23.334256 7681 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:23.334699 7681 webserver.cc:460] Webserver started at http://127.5.151.194:40311/ using document root <none> and password file <none>
I20260927 18:34:23.334932 7681 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:23.334995 7681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:23.335140 7681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:23.336143 7681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "0ee5dbc4643e45daa85a8933de54e13a"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "54dcb7564bb6abe074f9c10d3ec9c0ea"
server_key_iv: "5aa2776a9827c61f591d2d8cbd26b1a5"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.336488 7681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "0ee5dbc4643e45daa85a8933de54e13a"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "54dcb7564bb6abe074f9c10d3ec9c0ea"
server_key_iv: "5aa2776a9827c61f591d2d8cbd26b1a5"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.337764 7681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:23.338513 7696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.338683 7681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:23.338790 7681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "0ee5dbc4643e45daa85a8933de54e13a"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "54dcb7564bb6abe074f9c10d3ec9c0ea"
server_key_iv: "5aa2776a9827c61f591d2d8cbd26b1a5"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.338863 7681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:23.351215 7681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:23.353144 7681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:23.353310 7681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:23.353797 7681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:23.353856 7681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.353904 7681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:23.353932 7681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.360798 7681 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:43011
I20260927 18:34:23.360879 7811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:43011 every 8 connection(s)
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:23.361267 7681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:23.364127 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.354330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:41553 (local address 127.5.151.254:34949)
0927 18:34:23.354405 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.354408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.354571 (+ 163us) server_negotiation.cc:407] Connection header received
0927 18:34:23.355079 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.355081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.355096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.355173 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.355811 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.356620 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.358274 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.358650 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.358745 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.361541 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.361543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.361544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.361552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.362787 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.363207 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.363208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.363208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.363265 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.363504 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.363505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.363505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.363623 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.363697 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.363958 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.364035 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:34:23.364327 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.354265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34949 (local address 127.5.151.194:41553)
0927 18:34:23.354426 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.354465 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.354980 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.355263 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.355267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.355548 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.355730 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.355735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.356754 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.356756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.358167 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.358172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.358755 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.358757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.358944 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.359574 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.359586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.361449 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.362892 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.362899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.362901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.363121 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.363330 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.363331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.363331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.363441 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.363686 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.363693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.363831 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.363936 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.364092 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260927 18:34:23.364776 7812 heartbeater.cc:347] Connected to a master server at 127.5.151.254:34949
I20260927 18:34:23.364871 7812 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:23.365041 7812 heartbeater.cc:511] Master 127.5.151.254:34949 requested a full tablet report, sending...
I20260927 18:34:23.365456 7469 ts_manager.cc:194] Registered new tserver with Master: 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011)
I20260927 18:34:23.365913 7469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:41553
I20260927 18:34:23.367164 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 7681
I20260927 18:34:23.367275 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:23.367449 5727 external_mini_cluster.cc:1428] Setting key 7ef69d7c619c81ca5ed3eb2714e3eac0
I20260927 18:34:23.371747 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.365858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:58121 (local address 127.5.151.254:34949)
0927 18:34:23.365895 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.365897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.365970 (+ 73us) server_negotiation.cc:407] Connection header received
0927 18:34:23.366075 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.366076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.366087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.366136 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.366471 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.367028 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.367890 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.368159 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.368249 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.369059 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.369061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.369061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.369069 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.370506 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.370905 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.370906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.370907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.370959 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.371150 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.371151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.371152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.371328 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.371399 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.371565 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.371649 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:23.371757 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.365773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34949 (local address 127.5.151.194:58121)
0927 18:34:23.365901 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.365922 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.365987 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.366142 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.366142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.366335 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.366391 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.366392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.367124 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.367124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.367830 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.367833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.368238 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.368239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.368294 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.368711 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.368715 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.368981 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.370643 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.370644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.370644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.370822 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.371026 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.371027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.371027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.371085 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.371431 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.371432 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.371470 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.371539 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.371645 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:23.390842 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34949
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:23.470898 7821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:23.471117 7821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:23.471155 7821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:23.472299 7821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:23.472361 7821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:23.472394 7821 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:34:23.472421 7821 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:23.472452 7821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:23.473901 7821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:34949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:23.474176 7821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:23.474398 7821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:23.479221 7826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:23.479210 7827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:23.479290 7821 server_base.cc:1034] running on GCE node
W20260927 18:34:23.479207 7829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:23.479642 7821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:23.479876 7821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:23.481022 7821 hybrid_clock.cc:648] HybridClock initialized: now 1790534063481008 us; error 27 us; skew 500 ppm
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:23.483426 7821 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:23.483821 7821 webserver.cc:460] Webserver started at http://127.5.151.195:37791/ using document root <none> and password file <none>
I20260927 18:34:23.484014 7821 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:23.484053 7821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:23.484138 7821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:23.485108 7821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "b085150eccc94c51b92eed015f83ed51"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "9fc8fa37e34c4dad243822cf4fbd665c"
server_key_iv: "60de4a7a6c3e9906e598e19323d5ca57"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.485419 7821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "b085150eccc94c51b92eed015f83ed51"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "9fc8fa37e34c4dad243822cf4fbd665c"
server_key_iv: "60de4a7a6c3e9906e598e19323d5ca57"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.486680 7821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:23.487382 7836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.487605 7821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:23.487689 7821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "b085150eccc94c51b92eed015f83ed51"
format_stamp: "Formatted at 2026-09-27 18:34:23 on dist-test-slave-19nl"
server_key: "9fc8fa37e34c4dad243822cf4fbd665c"
server_key_iv: "60de4a7a6c3e9906e598e19323d5ca57"
server_key_version: "encryptionkey@0"
I20260927 18:34:23.487759 7821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:23.505388 7821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:23.507304 7821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:23.507467 7821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:23.507942 7821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:23.507982 7821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.508003 7821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:23.508049 7821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.514581 7821 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:35393
I20260927 18:34:23.514740 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:35393 every 8 connection(s)
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:23.515027 7821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:23.515265 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 7821
I20260927 18:34:23.515357 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:23.515540 5727 external_mini_cluster.cc:1428] Setting key b5e2d01dc96667870e1208e565974c76
I20260927 18:34:23.518219 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.508405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:41089 (local address 127.5.151.254:34949)
0927 18:34:23.508476 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.508479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.508659 (+ 180us) server_negotiation.cc:407] Connection header received
0927 18:34:23.509138 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.509140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.509153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.509230 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.510093 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.510936 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.512473 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.512811 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.512912 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.515456 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.515459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.515459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.515469 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.516953 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.517299 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.517300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.517300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.517374 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.517616 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.517617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.517617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.517730 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.517782 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.518034 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.518094 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:34:23.518369 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.508370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34949 (local address 127.5.151.195:41089)
0927 18:34:23.508534 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.508573 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.509036 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.509317 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.509321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.509668 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.509965 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.509972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.511048 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.511049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.512383 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.512385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.512911 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.512911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.513014 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.513531 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.513538 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.515370 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.517021 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.517026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.517028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.517239 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.517435 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.517436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.517436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.517536 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.517782 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.517785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.517894 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.518017 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.518165 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260927 18:34:23.518831 7952 heartbeater.cc:347] Connected to a master server at 127.5.151.254:34949
I20260927 18:34:23.518923 7952 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:23.519127 7952 heartbeater.cc:511] Master 127.5.151.254:34949 requested a full tablet report, sending...
I20260927 18:34:23.519532 7469 ts_manager.cc:194] Registered new tserver with Master: b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:23.520016 7469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:41089
I20260927 18:34:23.524878 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.519982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:45609 (local address 127.5.151.254:34949)
0927 18:34:23.520016 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.520018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.520234 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:34:23.520311 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.520312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.520322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.520391 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.520658 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.521150 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.521779 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.521955 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.522004 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.522776 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.522777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.522777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.522782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.524100 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.524376 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.524376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.524377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.524416 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.524549 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.524549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.524549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.524625 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.524667 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.524779 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.524826 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:34:23.524891 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.519871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34949 (local address 127.5.151.195:45609)
0927 18:34:23.520171 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.520182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.520243 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.520400 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.520400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.520543 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.520594 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.520595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.521225 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.521226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.521720 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.521721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.522019 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.522020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.522071 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.522380 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.522384 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.522709 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.524180 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.524181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.524181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.524320 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.524461 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.524461 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.524461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.524506 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.524667 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.524667 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.524718 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.524765 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.524830 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":60,"threads_started":1}
I20260927 18:34:23.527205 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:23.548738 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.542386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.5.151.254:34949)
0927 18:34:23.542444 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.542446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.542456 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:23.542511 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.542512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.542519 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.542636 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.543154 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.543623 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.544459 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.544634 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.544688 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.546785 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.546786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.546786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.546792 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.547872 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.548146 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.548146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.548146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.548192 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.548337 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.548337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.548337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.548448 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.548518 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.548589 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.548661 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:23.550177 7469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:23.551787 7469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:23.557714 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.554020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.5.151.193:42027)
0927 18:34:23.554211 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.554215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.554225 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:23.554265 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.554267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.554283 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.554365 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:23.554578 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.555647 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.556835 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.557301 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.557379 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.557436 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.557589 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.557624 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260927 18:34:23.557703 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.553778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42027 (local address 127.0.0.1:40572)
0927 18:34:23.553902 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.553941 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.554004 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.554435 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.554436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.554438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:23.554480 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.554481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.555731 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.555732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.556765 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.556767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.557390 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.557391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.557468 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.557481 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.557570 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.557621 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":82,"threads_started":1}
I20260927 18:34:23.557785 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.553788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35393 (local address 127.0.0.1:47648)
0927 18:34:23.554208 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.554215 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.554263 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.554597 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.554598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.554600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:23.554644 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.554646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.555452 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.555453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.556668 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.556670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.557518 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.557519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.557594 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.557599 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.557645 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.557691 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":71,"thread_start_us":70,"threads_started":1}
I20260927 18:34:23.558027 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.553970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.5.151.194:43011)
0927 18:34:23.554155 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.554160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.554170 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:23.554210 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.554212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.554230 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.554300 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:23.554515 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.555653 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.556634 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.557358 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.557428 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.557481 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.557887 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.557931 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260927 18:34:23.558122 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.554029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.5.151.195:35393)
0927 18:34:23.554210 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.554215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.554255 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:34:23.554365 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.554368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.554383 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.554515 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:23.554727 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.555368 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.556747 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.557385 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.557438 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.557481 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.557734 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.557964 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260927 18:34:23.558184 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.553780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43011 (local address 127.0.0.1:43098)
0927 18:34:23.554101 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.554110 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.554146 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.554336 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.554355 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.554358 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:23.554426 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.554429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.555748 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.555750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.556550 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.556552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.557516 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.557518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.557594 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.557598 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.557645 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.557691 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":8,"thread_start_us":71,"threads_started":1}
I20260927 18:34:23.558413 7606 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea545ae51304cf7bc881aa9ae45561e (DEFAULT_TABLE table=test-table [id=c884448bfa8746bcabd3fdb879327489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:23.558795 7606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea545ae51304cf7bc881aa9ae45561e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.558791 7886 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea545ae51304cf7bc881aa9ae45561e (DEFAULT_TABLE table=test-table [id=c884448bfa8746bcabd3fdb879327489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:23.559109 7886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea545ae51304cf7bc881aa9ae45561e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.561297 7972 tablet_bootstrap.cc:492] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Bootstrap starting.
I20260927 18:34:23.561560 7973 tablet_bootstrap.cc:492] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Bootstrap starting.
I20260927 18:34:23.562144 7973 tablet_bootstrap.cc:654] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.562151 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea545ae51304cf7bc881aa9ae45561e (DEFAULT_TABLE table=test-table [id=c884448bfa8746bcabd3fdb879327489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:23.562389 7972 tablet_bootstrap.cc:654] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.562472 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea545ae51304cf7bc881aa9ae45561e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.562485 7973 log.cc:824] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:23.562748 7972 log.cc:824] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:23.563205 7973 tablet_bootstrap.cc:492] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: No bootstrap required, opened a new log
I20260927 18:34:23.563292 7973 ts_tablet_manager.cc:1397] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:23.564523 7973 raft_consensus.cc:348] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.564685 7973 raft_consensus.cc:374] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.564728 7973 raft_consensus.cc:729] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e19f88b404f32a41a8b9bb3e43780, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.564827 7973 consensus_queue.cc:260] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.565099 7973 ts_tablet_manager.cc:1428] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:23.565169 7672 heartbeater.cc:503] Master 127.5.151.254:34949 was elected leader, sending a full tablet report...
I20260927 18:34:23.565622 7972 tablet_bootstrap.cc:492] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: No bootstrap required, opened a new log
I20260927 18:34:23.565694 7972 ts_tablet_manager.cc:1397] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 18:34:23.565794 7977 tablet_bootstrap.cc:492] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Bootstrap starting.
I20260927 18:34:23.566346 7977 tablet_bootstrap.cc:654] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.566623 7977 log.cc:824] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:23.567205 7977 tablet_bootstrap.cc:492] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: No bootstrap required, opened a new log
I20260927 18:34:23.567252 7977 ts_tablet_manager.cc:1397] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:34:23.567453 7972 raft_consensus.cc:348] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.567585 7972 raft_consensus.cc:374] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.567623 7972 raft_consensus.cc:729] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b085150eccc94c51b92eed015f83ed51, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.567735 7972 consensus_queue.cc:260] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.567981 7972 ts_tablet_manager.cc:1428] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:23.568039 7952 heartbeater.cc:503] Master 127.5.151.254:34949 was elected leader, sending a full tablet report...
I20260927 18:34:23.568867 7977 raft_consensus.cc:348] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.568992 7977 raft_consensus.cc:374] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.569013 7977 raft_consensus.cc:729] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.569105 7977 consensus_queue.cc:260] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.569311 7977 ts_tablet_manager.cc:1428] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:34:23.569383 7812 heartbeater.cc:503] Master 127.5.151.254:34949 was elected leader, sending a full tablet report...
W20260927 18:34:23.611886 7813 tablet.cc:2369] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:23.614228 7979 raft_consensus.cc:482] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:23.614324 7979 raft_consensus.cc:504] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.614754 7979 leader_election.cc:288] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011)
I20260927 18:34:23.617327 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.614934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43011 (local address 127.5.151.195:58633)
0927 18:34:23.614988 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.615000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.615041 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.615157 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.615158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.615159 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:23.615228 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.615229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.615828 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.615829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.616511 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.616513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.617042 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.617043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.617125 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.617128 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.617214 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.617250 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:34:23.617327 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.615012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:58633 (local address 127.5.151.194:43011)
0927 18:34:23.615050 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.615052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.615062 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:23.615090 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.615091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.615100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.615151 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:23.615290 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.615747 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.616614 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.616958 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.617035 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.617044 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.617222 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.617259 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:23.617470 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.615003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:50443 (local address 127.5.151.193:42027)
0927 18:34:23.615045 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.615047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.615158 (+ 111us) server_negotiation.cc:407] Connection header received
0927 18:34:23.615226 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.615227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.615235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.615298 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:23.615456 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.615899 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.616760 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.617120 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.617214 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.617222 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.617374 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.617408 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:23.617480 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.614935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42027 (local address 127.5.151.195:50443)
0927 18:34:23.615090 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.615099 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.615172 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.615299 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.615300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.615302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:23.615381 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.615382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.615978 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.615979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.616674 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.616675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.617212 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.617213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.617288 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.617291 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.617354 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.617395 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":11,"thread_start_us":83,"threads_started":1}
I20260927 18:34:23.617682 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea545ae51304cf7bc881aa9ae45561e" candidate_uuid: "b085150eccc94c51b92eed015f83ed51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee5dbc4643e45daa85a8933de54e13a" is_pre_election: true
I20260927 18:34:23.617766 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea545ae51304cf7bc881aa9ae45561e" candidate_uuid: "b085150eccc94c51b92eed015f83ed51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" is_pre_election: true
I20260927 18:34:23.617911 7766 raft_consensus.cc:2454] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b085150eccc94c51b92eed015f83ed51 in term 0.
I20260927 18:34:23.617935 7626 raft_consensus.cc:2454] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b085150eccc94c51b92eed015f83ed51 in term 0.
I20260927 18:34:23.618142 7837 leader_election.cc:302] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b085150eccc94c51b92eed015f83ed51, cb6e19f88b404f32a41a8b9bb3e43780; no voters:
I20260927 18:34:23.618253 7979 raft_consensus.cc:2781] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:23.618309 7979 raft_consensus.cc:482] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:23.618324 7979 raft_consensus.cc:3037] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:23.619262 7979 raft_consensus.cc:504] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.619393 7979 leader_election.cc:288] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [CANDIDATE]: Term 1 election: Requested vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011)
I20260927 18:34:23.619526 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea545ae51304cf7bc881aa9ae45561e" candidate_uuid: "b085150eccc94c51b92eed015f83ed51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee5dbc4643e45daa85a8933de54e13a"
I20260927 18:34:23.619565 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea545ae51304cf7bc881aa9ae45561e" candidate_uuid: "b085150eccc94c51b92eed015f83ed51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780"
I20260927 18:34:23.619620 7766 raft_consensus.cc:3037] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:23.619628 7626 raft_consensus.cc:3037] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:23.620507 7766 raft_consensus.cc:2454] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b085150eccc94c51b92eed015f83ed51 in term 1.
I20260927 18:34:23.620507 7626 raft_consensus.cc:2454] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b085150eccc94c51b92eed015f83ed51 in term 1.
I20260927 18:34:23.620671 7837 leader_election.cc:302] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b085150eccc94c51b92eed015f83ed51, cb6e19f88b404f32a41a8b9bb3e43780; no voters:
I20260927 18:34:23.620782 7979 raft_consensus.cc:2781] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:23.620915 7979 raft_consensus.cc:686] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 1 LEADER]: Becoming Leader. State: Replica: b085150eccc94c51b92eed015f83ed51, State: Running, Role: LEADER
I20260927 18:34:23.621124 7979 consensus_queue.cc:237] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.621937 7469 catalog_manager.cc:5495] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b085150eccc94c51b92eed015f83ed51 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "b085150eccc94c51b92eed015f83ed51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } health_report { overall_health: HEALTHY } } }
Sep 27 18:34:23 dist-test-slave-19nl krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:23.654106 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.647282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34949 (local address 127.0.0.1:56696)
0927 18:34:23.647483 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.647493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.647573 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.647719 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.647720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.648046 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:23.648087 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.648090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.648727 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.648729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.649276 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.649277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.649751 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.649752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.649814 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.650357 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:23.650361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:23.651715 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:23.653129 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.653135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.653137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.653326 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.653521 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:23.653521 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:23.653522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:23.653584 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:23.653801 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:23.653807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:23.653866 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.653922 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.654012 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
I20260927 18:34:23.654184 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.647365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.5.151.254:34949)
0927 18:34:23.647436 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.647438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.647553 (+ 115us) server_negotiation.cc:407] Connection header received
0927 18:34:23.647646 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.647647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.647660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.647714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:23.648180 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.648655 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.649472 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.649669 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.649751 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.651828 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:23.651829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:23.651830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:23.651836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:23.653063 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.653389 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.653390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.653390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.653437 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:23.653635 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:23.653636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:23.653636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:23.653736 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:23.653806 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.653950 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.654057 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":43}
I20260927 18:34:23.656136 7469 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:56696:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:34:23.656471 7469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:23.657900 7606 tablet_service.cc:1467] Processing CreateTablet for tablet 3c327cb5565a438eb69552ff2af6da07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0277b5c83e1d46c692d8c86dd3af5582]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:23.657913 7886 tablet_service.cc:1467] Processing CreateTablet for tablet 3c327cb5565a438eb69552ff2af6da07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0277b5c83e1d46c692d8c86dd3af5582]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:23.657886 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 3c327cb5565a438eb69552ff2af6da07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0277b5c83e1d46c692d8c86dd3af5582]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:23.658049 7886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c327cb5565a438eb69552ff2af6da07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.658049 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c327cb5565a438eb69552ff2af6da07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.658066 7606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c327cb5565a438eb69552ff2af6da07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.659399 7973 tablet_bootstrap.cc:492] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: Bootstrap starting.
I20260927 18:34:23.659574 7977 tablet_bootstrap.cc:492] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: Bootstrap starting.
I20260927 18:34:23.659602 7972 tablet_bootstrap.cc:492] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: Bootstrap starting.
I20260927 18:34:23.660158 7977 tablet_bootstrap.cc:654] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.660171 7973 tablet_bootstrap.cc:654] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.660171 7972 tablet_bootstrap.cc:654] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.660815 7977 tablet_bootstrap.cc:492] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: No bootstrap required, opened a new log
I20260927 18:34:23.660854 7973 tablet_bootstrap.cc:492] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: No bootstrap required, opened a new log
I20260927 18:34:23.660887 7977 ts_tablet_manager.cc:1397] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:23.660902 7973 ts_tablet_manager.cc:1397] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:23.661003 7972 tablet_bootstrap.cc:492] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: No bootstrap required, opened a new log
I20260927 18:34:23.661052 7972 ts_tablet_manager.cc:1397] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:23.661056 7977 raft_consensus.cc:348] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.661062 7973 raft_consensus.cc:348] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.661142 7973 raft_consensus.cc:374] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.661139 7977 raft_consensus.cc:374] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.661170 7973 raft_consensus.cc:729] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e19f88b404f32a41a8b9bb3e43780, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.661170 7977 raft_consensus.cc:729] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.661199 7972 raft_consensus.cc:348] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.661219 7977 consensus_queue.cc:260] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.661216 7973 consensus_queue.cc:260] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.661267 7972 raft_consensus.cc:374] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.661309 7972 raft_consensus.cc:729] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b085150eccc94c51b92eed015f83ed51, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.661312 7977 ts_tablet_manager.cc:1428] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:23.661324 7973 ts_tablet_manager.cc:1428] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:23.661352 7972 consensus_queue.cc:260] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.661444 7972 ts_tablet_manager.cc:1428] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.661417 7976 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.661456 7980 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.661510 7976 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:23.661520 7980 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:23.661490 7979 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.661556 7979 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 18:34:23.717326 7673 tablet.cc:2369] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:23.766058 7953 tablet.cc:2369] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:23.931170 7980 raft_consensus.cc:482] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:23.931279 7980 raft_consensus.cc:504] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.931656 7980 leader_election.cc:288] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:23.934477 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.931918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:60323 (local address 127.5.151.195:35393)
0927 18:34:23.931961 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.931963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.932051 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:34:23.932120 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.932122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.932132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.932181 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:23.932347 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.932835 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.933788 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.934125 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.934179 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.934203 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.934369 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.934410 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:34:23.934481 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.931895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:39461 (local address 127.5.151.193:42027)
0927 18:34:23.931943 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.931946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.932085 (+ 139us) server_negotiation.cc:407] Connection header received
0927 18:34:23.932142 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.932143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.932151 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.932218 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:23.932368 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.932829 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.933792 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.934139 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.934223 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.934230 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.934364 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.934400 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:34:23.934510 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.931830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35393 (local address 127.5.151.194:60323)
0927 18:34:23.931975 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.931985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.932044 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.932199 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.932200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.932203 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:23.932284 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.932286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.932935 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.932936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.933703 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.933705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.934203 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.934204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.934278 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.934287 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.934362 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.934403 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":5,"thread_start_us":71,"threads_started":1}
I20260927 18:34:23.934535 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.931833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42027 (local address 127.5.151.194:39461)
0927 18:34:23.931993 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.932002 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.932081 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.932230 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.932230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.932232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:23.932289 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.932291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.932909 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.932909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.933701 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.933703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.934231 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.934232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.934285 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.934288 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.934345 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.934388 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":28,"thread_start_us":64,"threads_started":1}
I20260927 18:34:23.934832 7906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c327cb5565a438eb69552ff2af6da07" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51" is_pre_election: true
I20260927 18:34:23.934932 7906 raft_consensus.cc:2454] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 0.
I20260927 18:34:23.934968 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c327cb5565a438eb69552ff2af6da07" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" is_pre_election: true
I20260927 18:34:23.935062 7626 raft_consensus.cc:2454] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 0.
I20260927 18:34:23.935120 7700 leader_election.cc:302] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, b085150eccc94c51b92eed015f83ed51; no voters:
I20260927 18:34:23.935246 7980 raft_consensus.cc:2781] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:23.935302 7980 raft_consensus.cc:482] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:23.935316 7980 raft_consensus.cc:3037] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:23.936018 7980 raft_consensus.cc:504] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.936146 7980 leader_election.cc:288] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 election: Requested vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:23.936290 7906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c327cb5565a438eb69552ff2af6da07" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51"
I20260927 18:34:23.936291 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c327cb5565a438eb69552ff2af6da07" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780"
I20260927 18:34:23.936358 7906 raft_consensus.cc:3037] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:23.936373 7626 raft_consensus.cc:3037] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:23.936983 7906 raft_consensus.cc:2454] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 1.
I20260927 18:34:23.937054 7626 raft_consensus.cc:2454] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 1.
I20260927 18:34:23.937131 7700 leader_election.cc:302] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, b085150eccc94c51b92eed015f83ed51; no voters:
I20260927 18:34:23.937202 7980 raft_consensus.cc:2781] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:23.937306 7980 raft_consensus.cc:686] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 LEADER]: Becoming Leader. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Running, Role: LEADER
I20260927 18:34:23.937413 7980 consensus_queue.cc:237] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.937798 7995 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: TxnStatusTablet state changed. Reason: New leader 0ee5dbc4643e45daa85a8933de54e13a. Latest consensus state: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.937891 7995 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:23.938071 7997 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:34:23.938196 7997 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:34:23.938186 7467 catalog_manager.cc:5495] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194). New cstate: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20260927 18:34:23.946380 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.944036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.5.151.194:43011)
0927 18:34:23.944078 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.944080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.944088 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:23.944191 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.944192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.944208 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.944259 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:23.944381 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.944925 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.945630 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.945832 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.945914 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.946084 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:23.946178 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:23.946254 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.946300 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.946320 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:34:23.946426 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.943982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43011 (local address 127.0.0.1:43104)
0927 18:34:23.944046 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.944059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.944109 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.944273 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.944274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.944275 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:23.944323 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.944324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.945002 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.945003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.945554 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.945555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.945916 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.945917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.945981 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.945994 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:23.946245 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:23.946253 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.946298 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.946344 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 18:34:23.947914 7906 raft_consensus.cc:1261] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Refusing update from remote peer 0ee5dbc4643e45daa85a8933de54e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:23.947988 7626 raft_consensus.cc:1261] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Refusing update from remote peer 0ee5dbc4643e45daa85a8933de54e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:23.948175 7980 consensus_queue.cc:1035] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:23.948258 7995 consensus_queue.cc:1035] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:23.949373 7979 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: TxnStatusTablet state changed. Reason: New leader 0ee5dbc4643e45daa85a8933de54e13a. Latest consensus state: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.949460 7979 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:23.950433 7976 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: TxnStatusTablet state changed. Reason: New leader 0ee5dbc4643e45daa85a8933de54e13a. Latest consensus state: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.950521 7976 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:23.950838 8000 mvcc.cc:205] Tried to move back new op lower bound from 7334027525928501248 to 7334027525888163840. Current Snapshot: MvccSnapshot[applied={T|T < 7334027525928501248}]
I20260927 18:34:23.950938 7995 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.951009 7995 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:23.951109 7980 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.951191 7980 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P 0ee5dbc4643e45daa85a8933de54e13a: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:23.951620 7979 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.951701 7979 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P b085150eccc94c51b92eed015f83ed51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:23.951803 8001 mvcc.cc:205] Tried to move back new op lower bound from 7334027525928501248 to 7334027525888163840. Current Snapshot: MvccSnapshot[applied={T|T < 7334027525928501248}]
I20260927 18:34:23.952615 7976 tablet_replica.cc:439] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } } }
I20260927 18:34:23.952694 7976 tablet_replica.cc:442] T 3c327cb5565a438eb69552ff2af6da07 P cb6e19f88b404f32a41a8b9bb3e43780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:23.956189 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.953682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.5.151.195:35393)
0927 18:34:23.953716 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.953718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.953729 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:34:23.953797 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.953797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.953809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.953865 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:23.954002 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.954427 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.955567 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.955741 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.955791 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.955954 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:23.956035 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:23.956085 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.956130 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.956145 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:34:23.960847 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.958312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43011 (local address 127.5.151.195:41805)
0927 18:34:23.958383 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.958394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.958435 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.958675 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.958676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.958677 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:23.958720 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.958721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.959543 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.959544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.960079 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.960080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.960518 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.960519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.960562 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.960573 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:23.960745 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:23.960747 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.960773 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.960802 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 18:34:23.960873 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.958491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:41805 (local address 127.5.151.194:43011)
0927 18:34:23.958567 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.958569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.958579 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:23.958599 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.958600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.958609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.958670 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:23.958763 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.959476 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.960156 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.960450 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.960520 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.960653 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:23.960685 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:23.960732 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.960790 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.960808 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:34:23.963112 7626 raft_consensus.cc:1261] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Refusing update from remote peer b085150eccc94c51b92eed015f83ed51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:23.963197 7766 raft_consensus.cc:1261] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Refusing update from remote peer b085150eccc94c51b92eed015f83ed51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:23.963361 7982 consensus_queue.cc:1035] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:23.963459 7979 consensus_queue.cc:1035] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:23.965353 8003 mvcc.cc:205] Tried to move back new op lower bound from 7334027525990961152 to 7334027524592803840. Current Snapshot: MvccSnapshot[applied={T|T < 7334027525990961152}]
I20260927 18:34:23.972589 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.969513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35393 (local address 127.5.151.194:58869)
0927 18:34:23.969658 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:23.969669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:23.969723 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:23.969856 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:23.969856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:23.969857 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:23.969978 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.969979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.970908 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.970909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.971758 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:23.971760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.972219 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.972219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:23.972281 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.972293 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:23.972450 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:23.972452 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:23.972486 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 18:34:23.972518 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260927 18:34:23.972581 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:23.969579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:58869 (local address 127.5.151.195:35393)
0927 18:34:23.969621 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:23.969623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:23.969726 (+ 103us) server_negotiation.cc:407] Connection header received
0927 18:34:23.969772 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:23.969773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:23.969783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:23.969854 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:23.970072 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.970830 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.971828 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:23.972141 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:23.972205 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:23.972364 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:23.972393 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:23.972446 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:23.972491 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 18:34:23.972510 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:23.985183 7467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:23.985303 7467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:23.987032 7467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Sending DeleteTablet for 3 replicas of tablet 0ea545ae51304cf7bc881aa9ae45561e
I20260927 18:34:23.987432 7606 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea545ae51304cf7bc881aa9ae45561e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:23 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:40572
I20260927 18:34:23.988695 7467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:23.988844 7467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:23.989681 7886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea545ae51304cf7bc881aa9ae45561e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:23 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47648
I20260927 18:34:23.989928 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea545ae51304cf7bc881aa9ae45561e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:23 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:43098
I20260927 18:34:23.991559 7746 tablet_service.cc:1467] Processing CreateTablet for tablet fc2035bfcfbf4b7d9a9ea33347d45070 (DEFAULT_TABLE table=test-table [id=6e5ec24375e044cabf5d1e9fbe37332f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:23.991559 7606 tablet_service.cc:1467] Processing CreateTablet for tablet fc2035bfcfbf4b7d9a9ea33347d45070 (DEFAULT_TABLE table=test-table [id=6e5ec24375e044cabf5d1e9fbe37332f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:23.991684 7606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2035bfcfbf4b7d9a9ea33347d45070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.991684 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2035bfcfbf4b7d9a9ea33347d45070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.992990 7977 tablet_bootstrap.cc:492] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: Bootstrap starting.
I20260927 18:34:23.993480 7977 tablet_bootstrap.cc:654] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.993883 7973 tablet_bootstrap.cc:492] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: Bootstrap starting.
I20260927 18:34:23.994253 7973 tablet_bootstrap.cc:654] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.994817 7886 tablet_service.cc:1467] Processing CreateTablet for tablet fc2035bfcfbf4b7d9a9ea33347d45070 (DEFAULT_TABLE table=test-table [id=6e5ec24375e044cabf5d1e9fbe37332f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:23.994941 7886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2035bfcfbf4b7d9a9ea33347d45070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:23.995930 7972 tablet_bootstrap.cc:492] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: Bootstrap starting.
I20260927 18:34:23.996300 7972 tablet_bootstrap.cc:654] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:23.997043 7973 tablet_bootstrap.cc:492] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: No bootstrap required, opened a new log
I20260927 18:34:23.997093 7973 ts_tablet_manager.cc:1397] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:34:23.997244 7973 raft_consensus.cc:348] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.997319 7973 raft_consensus.cc:374] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.997346 7973 raft_consensus.cc:729] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e19f88b404f32a41a8b9bb3e43780, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.997382 7973 consensus_queue.cc:260] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.997473 7973 ts_tablet_manager.cc:1428] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:23.997681 7972 tablet_bootstrap.cc:492] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: No bootstrap required, opened a new log
I20260927 18:34:23.997731 7972 ts_tablet_manager.cc:1397] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:23.997871 7972 raft_consensus.cc:348] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.997957 7972 raft_consensus.cc:374] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.997985 7972 raft_consensus.cc:729] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b085150eccc94c51b92eed015f83ed51, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.998026 7972 consensus_queue.cc:260] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.998113 7972 ts_tablet_manager.cc:1428] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:23.998862 7977 tablet_bootstrap.cc:492] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: No bootstrap required, opened a new log
I20260927 18:34:23.998924 7977 ts_tablet_manager.cc:1397] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 18:34:23.999085 7977 raft_consensus.cc:348] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.999176 7977 raft_consensus.cc:374] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:23.999204 7977 raft_consensus.cc:729] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Initialized, Role: FOLLOWER
I20260927 18:34:23.999245 7977 consensus_queue.cc:260] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:23.999351 7977 ts_tablet_manager.cc:1428] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:23.999833 8020 tablet_replica.cc:330] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: stopping tablet replica
I20260927 18:34:23.999971 8020 raft_consensus.cc:2229] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:24.000092 8020 raft_consensus.cc:2258] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:24.000247 8021 tablet_replica.cc:330] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: stopping tablet replica
I20260927 18:34:24.000411 8021 raft_consensus.cc:2229] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:24.000483 8021 raft_consensus.cc:2258] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:24.000685 8019 tablet_replica.cc:330] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: stopping tablet replica
I20260927 18:34:24.000799 8019 raft_consensus.cc:2229] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:24.000876 8019 raft_consensus.cc:2258] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:24.001072 8021 ts_tablet_manager.cc:1905] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:24.001272 8019 ts_tablet_manager.cc:1905] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:24.002537 8020 ts_tablet_manager.cc:1905] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:24.002735 8021 ts_tablet_manager.cc:1918] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:24.002847 8021 log.cc:1196] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/0ea545ae51304cf7bc881aa9ae45561e
I20260927 18:34:24.002875 8019 ts_tablet_manager.cc:1918] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:24.002946 8019 log.cc:1196] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/0ea545ae51304cf7bc881aa9ae45561e
I20260927 18:34:24.003223 8019 ts_tablet_manager.cc:1939] T 0ea545ae51304cf7bc881aa9ae45561e P cb6e19f88b404f32a41a8b9bb3e43780: Deleting consensus metadata
I20260927 18:34:24.003321 8021 ts_tablet_manager.cc:1939] T 0ea545ae51304cf7bc881aa9ae45561e P 0ee5dbc4643e45daa85a8933de54e13a: Deleting consensus metadata
I20260927 18:34:24.003634 7453 catalog_manager.cc:4841] TS cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027): tablet 0ea545ae51304cf7bc881aa9ae45561e (table test-table [id=c884448bfa8746bcabd3fdb879327489]) successfully deleted
I20260927 18:34:24.003780 7454 catalog_manager.cc:4841] TS 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011): tablet 0ea545ae51304cf7bc881aa9ae45561e (table test-table [id=c884448bfa8746bcabd3fdb879327489]) successfully deleted
I20260927 18:34:24.004369 8020 ts_tablet_manager.cc:1918] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:24.004449 8020 log.cc:1196] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/0ea545ae51304cf7bc881aa9ae45561e
I20260927 18:34:24.004669 8020 ts_tablet_manager.cc:1939] T 0ea545ae51304cf7bc881aa9ae45561e P b085150eccc94c51b92eed015f83ed51: Deleting consensus metadata
I20260927 18:34:24.005023 7456 catalog_manager.cc:4841] TS b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393): tablet 0ea545ae51304cf7bc881aa9ae45561e (table test-table [id=c884448bfa8746bcabd3fdb879327489]) successfully deleted
I20260927 18:34:24.087814 7980 raft_consensus.cc:482] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:24.087918 7980 raft_consensus.cc:504] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:24.088068 7980 leader_election.cc:288] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:24.088296 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2035bfcfbf4b7d9a9ea33347d45070" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" is_pre_election: true
I20260927 18:34:24.088279 7906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2035bfcfbf4b7d9a9ea33347d45070" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51" is_pre_election: true
I20260927 18:34:24.088374 7906 raft_consensus.cc:2454] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 0.
I20260927 18:34:24.088374 7626 raft_consensus.cc:2454] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 0.
I20260927 18:34:24.088523 7700 leader_election.cc:302] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, b085150eccc94c51b92eed015f83ed51; no voters:
I20260927 18:34:24.088634 7980 raft_consensus.cc:2781] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:24.088662 7980 raft_consensus.cc:482] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:24.088680 7980 raft_consensus.cc:3037] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:24.089339 7980 raft_consensus.cc:504] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:24.089473 7980 leader_election.cc:288] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 election: Requested vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:24.089663 7906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2035bfcfbf4b7d9a9ea33347d45070" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51"
I20260927 18:34:24.089661 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2035bfcfbf4b7d9a9ea33347d45070" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780"
I20260927 18:34:24.089720 7906 raft_consensus.cc:3037] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:24.089720 7626 raft_consensus.cc:3037] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:24.090485 7906 raft_consensus.cc:2454] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 1.
I20260927 18:34:24.090582 7626 raft_consensus.cc:2454] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 1.
I20260927 18:34:24.090682 7700 leader_election.cc:302] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, b085150eccc94c51b92eed015f83ed51; no voters:
I20260927 18:34:24.090858 7980 raft_consensus.cc:2781] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:24.090909 7980 raft_consensus.cc:686] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 LEADER]: Becoming Leader. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Running, Role: LEADER
I20260927 18:34:24.090955 7980 consensus_queue.cc:237] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:24.091459 7467 catalog_manager.cc:5495] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194). New cstate: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20260927 18:34:24.120734 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:24.118035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.5.151.194:43011)
0927 18:34:24.118080 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:24.118082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:24.118136 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:34:24.118202 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:24.118203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:24.118216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:24.118261 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:24.118490 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.119144 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.120057 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.120277 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.120347 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:24.120490 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:24.120524 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:24.120569 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:24.120623 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 18:34:24.120642 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:24.124543 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:24.122537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:51487 (local address 127.5.151.194:43011)
0927 18:34:24.122674 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:24.122675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:24.122679 (+ 4us) server_negotiation.cc:407] Connection header received
0927 18:34:24.122691 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:24.122691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:24.122696 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:24.122745 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:24.122830 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.123243 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.123911 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.124093 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.124166 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:24.124281 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:24.124305 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:24.124351 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:24.124425 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 18:34:24.124449 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":126}
I20260927 18:34:24.124543 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:24.122513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43011 (local address 127.5.151.194:51487)
0927 18:34:24.122577 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:24.122587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:24.122621 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:24.122748 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:24.122749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:24.122750 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:24.122781 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:24.122783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.123312 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.123313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:24.123823 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:24.123825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:24.124166 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:24.124166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:24.124222 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:24.124227 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:24.124352 (+ 125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:24.124353 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 18:34:24.124422 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:34:24.124471 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":4}
I20260927 18:34:24.127122 7626 raft_consensus.cc:1261] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Refusing update from remote peer 0ee5dbc4643e45daa85a8933de54e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:24.127378 7995 consensus_queue.cc:1035] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:24.128275 7906 raft_consensus.cc:1261] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Refusing update from remote peer 0ee5dbc4643e45daa85a8933de54e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:24.128724 8011 consensus_queue.cc:1035] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:34:24.132687 8000 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:34:25.650902 7467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:25.651005 7467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:25.651736 7467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Sending DeleteTablet for 3 replicas of tablet fc2035bfcfbf4b7d9a9ea33347d45070
I20260927 18:34:25.652000 7606 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2035bfcfbf4b7d9a9ea33347d45070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:40572
I20260927 18:34:25.652083 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2035bfcfbf4b7d9a9ea33347d45070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:43098
I20260927 18:34:25.652220 7886 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2035bfcfbf4b7d9a9ea33347d45070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47648
I20260927 18:34:25.652369 8052 tablet_replica.cc:330] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: stopping tablet replica
I20260927 18:34:25.652431 8052 raft_consensus.cc:2229] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:25.652467 8052 raft_consensus.cc:2258] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.652701 8052 ts_tablet_manager.cc:1905] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.654078 8051 tablet_replica.cc:330] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: stopping tablet replica
I20260927 18:34:25.654170 8051 raft_consensus.cc:2229] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:25.654268 8051 raft_consensus.cc:2258] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.654495 8052 ts_tablet_manager.cc:1918] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:34:25.654507 8051 ts_tablet_manager.cc:1905] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.654587 8052 log.cc:1196] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/fc2035bfcfbf4b7d9a9ea33347d45070
I20260927 18:34:25.654798 8052 ts_tablet_manager.cc:1939] T fc2035bfcfbf4b7d9a9ea33347d45070 P b085150eccc94c51b92eed015f83ed51: Deleting consensus metadata
I20260927 18:34:25.655169 7456 catalog_manager.cc:4841] TS b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393): tablet fc2035bfcfbf4b7d9a9ea33347d45070 (table test-table [id=6e5ec24375e044cabf5d1e9fbe37332f]) successfully deleted
I20260927 18:34:25.655788 8051 ts_tablet_manager.cc:1918] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:34:25.655835 8051 log.cc:1196] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/fc2035bfcfbf4b7d9a9ea33347d45070
I20260927 18:34:25.656008 8051 ts_tablet_manager.cc:1939] T fc2035bfcfbf4b7d9a9ea33347d45070 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting consensus metadata
I20260927 18:34:25.656126 8050 tablet_replica.cc:330] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: stopping tablet replica
I20260927 18:34:25.656170 8050 raft_consensus.cc:2229] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:25.656234 8050 raft_consensus.cc:2258] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.656399 7454 catalog_manager.cc:4841] TS 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011): tablet fc2035bfcfbf4b7d9a9ea33347d45070 (table test-table [id=6e5ec24375e044cabf5d1e9fbe37332f]) successfully deleted
I20260927 18:34:25.656423 8050 ts_tablet_manager.cc:1905] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.657586 8050 ts_tablet_manager.cc:1918] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:34:25.657635 8050 log.cc:1196] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/fc2035bfcfbf4b7d9a9ea33347d45070
I20260927 18:34:25.657842 8050 ts_tablet_manager.cc:1939] T fc2035bfcfbf4b7d9a9ea33347d45070 P cb6e19f88b404f32a41a8b9bb3e43780: Deleting consensus metadata
I20260927 18:34:25.658157 7453 catalog_manager.cc:4841] TS cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027): tablet fc2035bfcfbf4b7d9a9ea33347d45070 (table test-table [id=6e5ec24375e044cabf5d1e9fbe37332f]) successfully deleted
Sep 27 18:34:25 dist-test-slave-19nl krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:34:25 dist-test-slave-19nl krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:25.683110 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.670801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.5.151.254:34949)
0927 18:34:25.670956 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.670958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.670994 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:34:25.671069 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.671070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.671088 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.671163 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:25.671697 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.672190 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.672931 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.673130 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.673206 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.680726 (+ 7520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:25.680728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:25.680728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:25.680735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:25.681912 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.682247 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.682249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.682249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.682309 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:25.682511 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:25.682512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:25.682512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:25.682685 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:25.682796 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.682900 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.682997 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260927 18:34:25.684311 7467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:25.684453 7467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:25.685664 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 5175103c44c246308805467822566aa2 (DEFAULT_TABLE table=test-table [id=4ea4d8649a154561a991ada7bad72d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.685673 7606 tablet_service.cc:1467] Processing CreateTablet for tablet 5175103c44c246308805467822566aa2 (DEFAULT_TABLE table=test-table [id=4ea4d8649a154561a991ada7bad72d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.685751 7886 tablet_service.cc:1467] Processing CreateTablet for tablet 5175103c44c246308805467822566aa2 (DEFAULT_TABLE table=test-table [id=4ea4d8649a154561a991ada7bad72d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.685791 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175103c44c246308805467822566aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.685796 7606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175103c44c246308805467822566aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.685884 7886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175103c44c246308805467822566aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.686970 8061 tablet_bootstrap.cc:492] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: Bootstrap starting.
I20260927 18:34:25.687381 8062 tablet_bootstrap.cc:492] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: Bootstrap starting.
I20260927 18:34:25.687400 8061 tablet_bootstrap.cc:654] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.687764 8062 tablet_bootstrap.cc:654] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.688043 8061 tablet_bootstrap.cc:492] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: No bootstrap required, opened a new log
I20260927 18:34:25.688089 8063 tablet_bootstrap.cc:492] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: Bootstrap starting.
I20260927 18:34:25.688112 8061 ts_tablet_manager.cc:1397] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:25.688237 8061 raft_consensus.cc:348] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.688319 8061 raft_consensus.cc:374] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.688349 8061 raft_consensus.cc:729] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.688340 8062 tablet_bootstrap.cc:492] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: No bootstrap required, opened a new log
I20260927 18:34:25.688383 8062 ts_tablet_manager.cc:1397] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:25.688408 8061 consensus_queue.cc:260] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.688477 8063 tablet_bootstrap.cc:654] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.688505 8062 raft_consensus.cc:348] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.688575 8062 raft_consensus.cc:374] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.688599 8062 raft_consensus.cc:729] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b085150eccc94c51b92eed015f83ed51, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.688651 8062 consensus_queue.cc:260] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.688829 8062 ts_tablet_manager.cc:1428] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:25.688947 8061 ts_tablet_manager.cc:1428] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:25.689543 8063 tablet_bootstrap.cc:492] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: No bootstrap required, opened a new log
I20260927 18:34:25.689603 8063 ts_tablet_manager.cc:1397] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:25.689774 8063 raft_consensus.cc:348] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.689849 8063 raft_consensus.cc:374] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.689864 8063 raft_consensus.cc:729] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e19f88b404f32a41a8b9bb3e43780, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.689917 8063 consensus_queue.cc:260] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.690133 8063 ts_tablet_manager.cc:1428] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:25.755388 8068 raft_consensus.cc:482] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:25.755472 8068 raft_consensus.cc:504] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.755822 8068 leader_election.cc:288] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:25.759025 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.755984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43011 (local address 127.5.151.193:41455)
0927 18:34:25.756182 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.756195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.756249 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.756412 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.756414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.756416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:25.756497 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.756499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.757160 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.757161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.757953 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.757955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.758747 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.758748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.758823 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.758826 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.758885 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.758926 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":18,"thread_start_us":104,"threads_started":1}
I20260927 18:34:25.759024 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.756074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:41455 (local address 127.5.151.194:43011)
0927 18:34:25.756249 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.756251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.756261 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:25.756317 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.756318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.756328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.756397 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:25.756573 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.757091 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.758074 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.758671 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.758743 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.758753 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.758905 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.758961 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260927 18:34:25.759286 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175103c44c246308805467822566aa2" candidate_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee5dbc4643e45daa85a8933de54e13a" is_pre_election: true
I20260927 18:34:25.759397 7766 raft_consensus.cc:2454] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6e19f88b404f32a41a8b9bb3e43780 in term 0.
I20260927 18:34:25.759537 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.755984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35393 (local address 127.5.151.193:49039)
0927 18:34:25.756182 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.756195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.756256 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.756475 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.756476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.756477 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:25.756555 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.756557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.757263 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.757264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.758458 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.758460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.759263 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.759264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.759340 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.759343 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.759404 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.759440 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":122,"threads_started":1}
I20260927 18:34:25.759598 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.756188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:49039 (local address 127.5.151.195:35393)
0927 18:34:25.756323 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.756325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.756340 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:34:25.756361 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.756361 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.756390 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.756456 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:25.756632 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.757183 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.758551 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.759039 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.759127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.759146 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.759468 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.759522 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260927 18:34:25.759581 7558 leader_election.cc:302] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, cb6e19f88b404f32a41a8b9bb3e43780; no voters:
I20260927 18:34:25.759703 8068 raft_consensus.cc:2781] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:25.759747 8068 raft_consensus.cc:482] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:25.759773 8068 raft_consensus.cc:3037] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.759850 7905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175103c44c246308805467822566aa2" candidate_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51" is_pre_election: true
I20260927 18:34:25.759954 7905 raft_consensus.cc:2454] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6e19f88b404f32a41a8b9bb3e43780 in term 0.
I20260927 18:34:25.760542 8068 raft_consensus.cc:504] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.760676 8068 leader_election.cc:288] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [CANDIDATE]: Term 1 election: Requested vote from peers 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:25.760807 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175103c44c246308805467822566aa2" candidate_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee5dbc4643e45daa85a8933de54e13a"
I20260927 18:34:25.760847 7905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175103c44c246308805467822566aa2" candidate_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51"
I20260927 18:34:25.760881 7766 raft_consensus.cc:3037] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.760921 7905 raft_consensus.cc:3037] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.761471 7766 raft_consensus.cc:2454] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6e19f88b404f32a41a8b9bb3e43780 in term 1.
I20260927 18:34:25.761574 7905 raft_consensus.cc:2454] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6e19f88b404f32a41a8b9bb3e43780 in term 1.
I20260927 18:34:25.761631 7558 leader_election.cc:302] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, cb6e19f88b404f32a41a8b9bb3e43780; no voters:
I20260927 18:34:25.761711 8068 raft_consensus.cc:2781] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:25.761835 8068 raft_consensus.cc:686] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 LEADER]: Becoming Leader. State: Replica: cb6e19f88b404f32a41a8b9bb3e43780, State: Running, Role: LEADER
I20260927 18:34:25.761940 8068 consensus_queue.cc:237] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.762609 7469 catalog_manager.cc:5495] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20260927 18:34:25.782677 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.780118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.5.151.193:42027)
0927 18:34:25.780270 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.780271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.780282 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:34:25.780310 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.780311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.780326 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.780405 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:25.780570 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.781031 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.781756 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.782042 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.782099 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.782302 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:25.782404 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:25.782490 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.782539 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.782582 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260927 18:34:25.787286 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.784930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:57689 (local address 127.5.151.194:43011)
0927 18:34:25.784975 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.784976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.785019 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:34:25.785082 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.785083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.785093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.785144 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:25.785264 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.785718 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.786498 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.786763 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.786845 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.787037 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:25.787076 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:25.787134 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.787196 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.787216 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:25.787319 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.784805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43011 (local address 127.5.151.193:57689)
0927 18:34:25.784945 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.784957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.785034 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.785152 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.785152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.785154 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:25.785205 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.785206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.785798 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.785799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.786407 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.786409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.786847 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.786848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.786918 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.786937 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:25.787136 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:25.787138 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.787191 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.787228 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260927 18:34:25.789163 7905 raft_consensus.cc:1261] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Refusing update from remote peer cb6e19f88b404f32a41a8b9bb3e43780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:25.789196 7766 raft_consensus.cc:1261] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Refusing update from remote peer cb6e19f88b404f32a41a8b9bb3e43780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:25.789412 8068 consensus_queue.cc:1035] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:25.789592 8073 consensus_queue.cc:1035] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:25.796005 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.793616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:60891 (local address 127.5.151.193:42027)
0927 18:34:25.793655 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.793657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.793745 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:34:25.793810 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.793811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.793821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.793872 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:25.794016 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.794511 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.795315 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.795610 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.795672 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.795837 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:25.795860 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:25.795901 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.795950 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.795964 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:25.796061 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.793579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42027 (local address 127.5.151.194:60891)
0927 18:34:25.793691 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:25.793701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:25.793759 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:25.793915 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:25.793915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:25.793917 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:25.793963 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.793964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.794608 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.794609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.795234 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:25.795236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.795681 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.795682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:25.795744 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.795749 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:25.795914 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:25.795916 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:25.795950 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 18:34:25.795983 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":73,"threads_started":1}
I20260927 18:34:25.805344 7467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:25.805446 7467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:25.806083 7467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Sending DeleteTablet for 3 replicas of tablet 5175103c44c246308805467822566aa2
I20260927 18:34:25.806308 7606 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175103c44c246308805467822566aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:40572
I20260927 18:34:25.806336 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175103c44c246308805467822566aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:43098
I20260927 18:34:25.806393 8050 tablet_replica.cc:330] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: stopping tablet replica
I20260927 18:34:25.806439 8050 raft_consensus.cc:2229] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:25.806452 8051 tablet_replica.cc:330] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: stopping tablet replica
I20260927 18:34:25.806447 7886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175103c44c246308805467822566aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:25 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47648
I20260927 18:34:25.806499 8051 raft_consensus.cc:2229] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:25.806505 8050 raft_consensus.cc:2258] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.806521 8052 tablet_replica.cc:330] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: stopping tablet replica
I20260927 18:34:25.806533 8051 raft_consensus.cc:2258] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.806584 8052 raft_consensus.cc:2229] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:25.806625 8052 raft_consensus.cc:2258] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:25.806882 8052 ts_tablet_manager.cc:1905] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.806896 8051 ts_tablet_manager.cc:1905] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.806970 8050 ts_tablet_manager.cc:1905] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:25.808527 8050 ts_tablet_manager.cc:1918] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:25.808590 8050 log.cc:1196] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/5175103c44c246308805467822566aa2
I20260927 18:34:25.808605 8051 ts_tablet_manager.cc:1918] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:25.808656 8051 log.cc:1196] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/5175103c44c246308805467822566aa2
I20260927 18:34:25.808604 7467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:25.808743 7467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:25.808810 8050 ts_tablet_manager.cc:1939] T 5175103c44c246308805467822566aa2 P cb6e19f88b404f32a41a8b9bb3e43780: Deleting consensus metadata
I20260927 18:34:25.808878 8051 ts_tablet_manager.cc:1939] T 5175103c44c246308805467822566aa2 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting consensus metadata
I20260927 18:34:25.809113 7453 catalog_manager.cc:4841] TS cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027): tablet 5175103c44c246308805467822566aa2 (table test-table [id=4ea4d8649a154561a991ada7bad72d95]) successfully deleted
I20260927 18:34:25.809268 7454 catalog_manager.cc:4841] TS 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011): tablet 5175103c44c246308805467822566aa2 (table test-table [id=4ea4d8649a154561a991ada7bad72d95]) successfully deleted
I20260927 18:34:25.809671 8052 ts_tablet_manager.cc:1918] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:25.809723 8052 log.cc:1196] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/5175103c44c246308805467822566aa2
I20260927 18:34:25.809924 8052 ts_tablet_manager.cc:1939] T 5175103c44c246308805467822566aa2 P b085150eccc94c51b92eed015f83ed51: Deleting consensus metadata
I20260927 18:34:25.810204 7456 catalog_manager.cc:4841] TS b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393): tablet 5175103c44c246308805467822566aa2 (table test-table [id=4ea4d8649a154561a991ada7bad72d95]) successfully deleted
I20260927 18:34:25.810441 7886 tablet_service.cc:1467] Processing CreateTablet for tablet 111f794bda584c8f9948540ce8b07270 (DEFAULT_TABLE table=test-table [id=5b26fa17729d43049f436b84a0ab0460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.810438 7606 tablet_service.cc:1467] Processing CreateTablet for tablet 111f794bda584c8f9948540ce8b07270 (DEFAULT_TABLE table=test-table [id=5b26fa17729d43049f436b84a0ab0460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.810438 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 111f794bda584c8f9948540ce8b07270 (DEFAULT_TABLE table=test-table [id=5b26fa17729d43049f436b84a0ab0460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:25.810539 7606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111f794bda584c8f9948540ce8b07270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.810539 7886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111f794bda584c8f9948540ce8b07270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.810537 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111f794bda584c8f9948540ce8b07270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:25.811655 8062 tablet_bootstrap.cc:492] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: Bootstrap starting.
I20260927 18:34:25.811669 8063 tablet_bootstrap.cc:492] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780: Bootstrap starting.
I20260927 18:34:25.811821 8061 tablet_bootstrap.cc:492] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: Bootstrap starting.
I20260927 18:34:25.811944 8062 tablet_bootstrap.cc:654] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.812085 8061 tablet_bootstrap.cc:654] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.812083 8063 tablet_bootstrap.cc:654] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:25.812562 8062 tablet_bootstrap.cc:492] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: No bootstrap required, opened a new log
I20260927 18:34:25.812618 8062 ts_tablet_manager.cc:1397] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:25.812731 8062 raft_consensus.cc:348] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.812810 8062 raft_consensus.cc:374] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.812834 8062 raft_consensus.cc:729] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b085150eccc94c51b92eed015f83ed51, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.812880 8062 consensus_queue.cc:260] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.812911 8061 tablet_bootstrap.cc:492] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: No bootstrap required, opened a new log
I20260927 18:34:25.812953 8061 ts_tablet_manager.cc:1397] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:25.813004 8062 ts_tablet_manager.cc:1428] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:25.813072 8063 tablet_bootstrap.cc:492] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780: No bootstrap required, opened a new log
I20260927 18:34:25.813047 8061 raft_consensus.cc:348] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.813108 8061 raft_consensus.cc:374] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.813112 8063 ts_tablet_manager.cc:1397] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:25.813128 8061 raft_consensus.cc:729] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.813166 8061 consensus_queue.cc:260] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.813246 8061 ts_tablet_manager.cc:1428] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:25.813271 8063 raft_consensus.cc:348] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.813344 8063 raft_consensus.cc:374] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:25.813366 8063 raft_consensus.cc:729] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e19f88b404f32a41a8b9bb3e43780, State: Initialized, Role: FOLLOWER
I20260927 18:34:25.813432 8063 consensus_queue.cc:260] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.813572 8063 ts_tablet_manager.cc:1428] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:25.848762 8038 raft_consensus.cc:482] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:25.848853 8038 raft_consensus.cc:504] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.848995 8038 leader_election.cc:288] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:25.849181 7906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111f794bda584c8f9948540ce8b07270" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51" is_pre_election: true
I20260927 18:34:25.849176 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111f794bda584c8f9948540ce8b07270" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" is_pre_election: true
I20260927 18:34:25.849267 7906 raft_consensus.cc:2454] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 0.
I20260927 18:34:25.849285 7626 raft_consensus.cc:2454] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 0.
I20260927 18:34:25.849411 7700 leader_election.cc:302] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, b085150eccc94c51b92eed015f83ed51; no voters:
I20260927 18:34:25.849524 8038 raft_consensus.cc:2781] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:25.849579 8038 raft_consensus.cc:482] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:25.849601 8038 raft_consensus.cc:3037] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.850418 8038 raft_consensus.cc:504] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.850648 8038 leader_election.cc:288] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 election: Requested vote from peers cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027), b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393)
I20260927 18:34:25.850729 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111f794bda584c8f9948540ce8b07270" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6e19f88b404f32a41a8b9bb3e43780"
I20260927 18:34:25.850797 7626 raft_consensus.cc:3037] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.850821 7906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111f794bda584c8f9948540ce8b07270" candidate_uuid: "0ee5dbc4643e45daa85a8933de54e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b085150eccc94c51b92eed015f83ed51"
I20260927 18:34:25.850885 7906 raft_consensus.cc:3037] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:25.851321 7626 raft_consensus.cc:2454] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 1.
I20260927 18:34:25.851490 7697 leader_election.cc:302] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5dbc4643e45daa85a8933de54e13a, cb6e19f88b404f32a41a8b9bb3e43780; no voters:
I20260927 18:34:25.851588 8038 raft_consensus.cc:2781] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:25.851584 7906 raft_consensus.cc:2454] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee5dbc4643e45daa85a8933de54e13a in term 1.
I20260927 18:34:25.851635 8038 raft_consensus.cc:686] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 LEADER]: Becoming Leader. State: Replica: 0ee5dbc4643e45daa85a8933de54e13a, State: Running, Role: LEADER
I20260927 18:34:25.851688 8038 consensus_queue.cc:237] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } }
I20260927 18:34:25.852141 7469 catalog_manager.cc:5495] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194). New cstate: current_term: 1 leader_uuid: "0ee5dbc4643e45daa85a8933de54e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee5dbc4643e45daa85a8933de54e13a" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20260927 18:34:25.862632 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:25.860171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.5.151.194:43011)
0927 18:34:25.860225 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:25.860227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:25.860235 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:25.860258 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:25.860258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:25.860269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:25.860330 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:25.860432 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.861167 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.861849 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:25.862155 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:25.862230 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:25.862348 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:25.862383 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:25.862459 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:25.862506 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 18:34:25.862528 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:34:25.864669 7906 raft_consensus.cc:1261] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Refusing update from remote peer 0ee5dbc4643e45daa85a8933de54e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:25.864857 7626 raft_consensus.cc:1261] T 111f794bda584c8f9948540ce8b07270 P cb6e19f88b404f32a41a8b9bb3e43780 [term 1 FOLLOWER]: Refusing update from remote peer 0ee5dbc4643e45daa85a8933de54e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:25.864902 8030 consensus_queue.cc:1035] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b085150eccc94c51b92eed015f83ed51" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:25.865049 8030 consensus_queue.cc:1035] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6e19f88b404f32a41a8b9bb3e43780" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:34:25.868923 8031 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:34:27.389518 7467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:27.389612 7467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:27.390407 7467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2617ac70abc4cb1b940efde4263189f: Sending DeleteTablet for 3 replicas of tablet 111f794bda584c8f9948540ce8b07270
I20260927 18:34:27.390658 7606 tablet_service.cc:1514] Processing DeleteTablet for tablet 111f794bda584c8f9948540ce8b07270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:27 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:40572
I20260927 18:34:27.390765 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 111f794bda584c8f9948540ce8b07270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:27 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:43098
I20260927 18:34:27.390902 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 7541
I20260927 18:34:27.391978 8114 tablet_replica.cc:330] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: stopping tablet replica
I20260927 18:34:27.392040 8114 raft_consensus.cc:2229] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:27.392099 8114 raft_consensus.cc:2258] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:27.392339 8114 ts_tablet_manager.cc:1905] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:27.390872 7886 tablet_service.cc:1514] Processing DeleteTablet for tablet 111f794bda584c8f9948540ce8b07270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:27 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47648
I20260927 18:34:27.393178 8115 tablet_replica.cc:330] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: stopping tablet replica
I20260927 18:34:27.393247 8115 raft_consensus.cc:2229] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:27.393333 8115 raft_consensus.cc:2258] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:27.393963 8114 ts_tablet_manager.cc:1918] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:34:27.394038 8114 log.cc:1196] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/111f794bda584c8f9948540ce8b07270
I20260927 18:34:27.394268 8114 ts_tablet_manager.cc:1939] T 111f794bda584c8f9948540ce8b07270 P 0ee5dbc4643e45daa85a8933de54e13a: Deleting consensus metadata
I20260927 18:34:27.394594 7454 catalog_manager.cc:4841] TS 0ee5dbc4643e45daa85a8933de54e13a (127.5.151.194:43011): tablet 111f794bda584c8f9948540ce8b07270 (table test-table [id=5b26fa17729d43049f436b84a0ab0460]) successfully deleted
I20260927 18:34:27.397435 8115 ts_tablet_manager.cc:1905] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:34:27.397547 7453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:42027 (error 108)
I20260927 18:34:27.397774 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 7681
I20260927 18:34:27.398931 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:27.397903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42027 (local address 127.0.0.1:40592)
0927 18:34:27.398854 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:27.398900 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:42027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":932,"thread_start_us":55,"threads_started":1}
W20260927 18:34:27.399031 7453 catalog_manager.cc:4584] TS cb6e19f88b404f32a41a8b9bb3e43780 (127.5.151.193:42027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 111f794bda584c8f9948540ce8b07270: Network error: Client connection negotiation failed: client connection to 127.5.151.193:42027: connect: Connection refused (error 111)
I20260927 18:34:27.399868 8115 ts_tablet_manager.cc:1918] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:34:27.399921 8115 log.cc:1196] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/111f794bda584c8f9948540ce8b07270
I20260927 18:34:27.400131 8115 ts_tablet_manager.cc:1939] T 111f794bda584c8f9948540ce8b07270 P b085150eccc94c51b92eed015f83ed51: Deleting consensus metadata
I20260927 18:34:27.400558 7456 catalog_manager.cc:4841] TS b085150eccc94c51b92eed015f83ed51 (127.5.151.195:35393): tablet 111f794bda584c8f9948540ce8b07270 (table test-table [id=5b26fa17729d43049f436b84a0ab0460]) successfully deleted
I20260927 18:34:27.404589 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 7821
I20260927 18:34:27.409570 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 7437
2026-09-27T18:34:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5676 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:27 dist-test-slave-19nl krb5kdc[8120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:27 dist-test-slave-19nl krb5kdc[8120](info): set up 2 sockets
Sep 27 18:34:27 dist-test-slave-19nl krb5kdc[8120](info): commencing operation
krb5kdc: starting...
W20260927 18:34:28.422272 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:28 dist-test-slave-19nl krb5kdc[8120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:28.530143 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41453
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:28.606163 8133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:28.606380 8133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:28.606416 8133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:28.607499 8133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:28.607560 8133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:28.607584 8133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:28.607610 8133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:28.607638 8133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:28.609020 8133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:28.609268 8133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:28.609480 8133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:28.613902 8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:28.614058 8133 server_base.cc:1034] running on GCE node
W20260927 18:34:28.613902 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:28.613924 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:28.614426 8133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:28.614717 8133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:28.615857 8133 hybrid_clock.cc:648] HybridClock initialized: now 1790534068615842 us; error 29 us; skew 500 ppm
Sep 27 18:34:28 dist-test-slave-19nl krb5kdc[8120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:28.618201 8133 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:28.618625 8133 webserver.cc:460] Webserver started at http://127.5.151.254:34735/ using document root <none> and password file <none>
I20260927 18:34:28.618821 8133 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:28.618862 8133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:28.618947 8133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:28.619894 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "df6c5669c4a247b98548adede1d7a687"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "42006f55b72975cfb21ad56798c4e94d"
server_key_iv: "23361715ddf4dedd86c7dc0efedc6fbd"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.620173 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "df6c5669c4a247b98548adede1d7a687"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "42006f55b72975cfb21ad56798c4e94d"
server_key_iv: "23361715ddf4dedd86c7dc0efedc6fbd"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.621379 8133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:28.622067 8148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:28.622215 8133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:28.622299 8133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "df6c5669c4a247b98548adede1d7a687"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "42006f55b72975cfb21ad56798c4e94d"
server_key_iv: "23361715ddf4dedd86c7dc0efedc6fbd"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.622363 8133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:28.635892 8133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:28.637521 8133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:28.637622 8133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:28.641218 8133 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:41453
I20260927 18:34:28.641275 8200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:41453 every 8 connection(s)
I20260927 18:34:28.641601 8133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:28.642433 8201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:28.644464 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8133
I20260927 18:34:28.644534 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:28.644665 8201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687: Bootstrap starting.
I20260927 18:34:28.644712 5727 external_mini_cluster.cc:1428] Setting key 682a457f9d035fe59830ff4db2eec367
I20260927 18:34:28.645393 8201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:28.645700 8201 log.cc:824] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:28.646421 8201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687: No bootstrap required, opened a new log
I20260927 18:34:28.648710 8201 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6c5669c4a247b98548adede1d7a687" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41453 } }
I20260927 18:34:28.648836 8201 raft_consensus.cc:374] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:28.648857 8201 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6c5669c4a247b98548adede1d7a687, State: Initialized, Role: FOLLOWER
I20260927 18:34:28.648991 8201 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6c5669c4a247b98548adede1d7a687" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41453 } }
I20260927 18:34:28.649063 8201 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:28.649102 8201 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:28.649149 8201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:28.649819 8201 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6c5669c4a247b98548adede1d7a687" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41453 } }
I20260927 18:34:28.649924 8201 leader_election.cc:302] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6c5669c4a247b98548adede1d7a687; no voters:
I20260927 18:34:28.650091 8201 leader_election.cc:288] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:28.650141 8206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:28.650353 8201 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:28.650357 8206 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [term 1 LEADER]: Becoming Leader. State: Replica: df6c5669c4a247b98548adede1d7a687, State: Running, Role: LEADER
I20260927 18:34:28.650475 8206 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6c5669c4a247b98548adede1d7a687" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41453 } }
I20260927 18:34:28.650823 8207 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df6c5669c4a247b98548adede1d7a687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6c5669c4a247b98548adede1d7a687" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41453 } } }
I20260927 18:34:28.650940 8207 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:28.651036 8208 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6c5669c4a247b98548adede1d7a687. Latest consensus state: current_term: 1 leader_uuid: "df6c5669c4a247b98548adede1d7a687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6c5669c4a247b98548adede1d7a687" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41453 } } }
I20260927 18:34:28.651146 8208 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:28.651270 8215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:28.651754 8215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 18:34:28 dist-test-slave-19nl krb5kdc[8120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:28.653586 8222 mvcc.cc:205] Tried to move back new op lower bound from 7334027545201418240 to 7334027545192763392. Current Snapshot: MvccSnapshot[applied={T|T < 7334027545201418240}]
I20260927 18:34:28.653972 8215 catalog_manager.cc:1347] Generated new cluster ID: a6b2f75a2f99422e82c7ee7dfb16bcdf
I20260927 18:34:28.654021 8215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:28.656698 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:28.645211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.5.151.254:41453)
0927 18:34:28.645400 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:28.645408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:28.645442 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:34:28.646039 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:28.646046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:28.646114 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:28.646265 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:28.646945 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:28.648398 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:28.649307 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:28.649919 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:28.650038 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:28.653026 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:28.653030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:28.653030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:28.653042 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:28.654992 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:28.655339 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:28.655342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:28.655342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:28.655434 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:28.655636 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:28.655637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:28.655638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:28.655963 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:28.656127 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:28.656233 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 18:34:28.656371 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260927 18:34:28.664970 8215 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:28.665603 8215 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:28.671679 8215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df6c5669c4a247b98548adede1d7a687: Generated new TSK 0
I20260927 18:34:28.671861 8215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:28.703612 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41453
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:28.801317 8227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:28.801543 8227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:28.801565 8227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:28.802695 8227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:28.802732 8227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:28.802758 8227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:28.804111 8227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:28.804340 8227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:28.804575 8227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:28.809541 8233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:28.809558 8227 server_base.cc:1034] running on GCE node
W20260927 18:34:28.809520 8235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:28.809531 8232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:28.809979 8227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:28.810218 8227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:28.811385 8227 hybrid_clock.cc:648] HybridClock initialized: now 1790534068811384 us; error 36 us; skew 500 ppm
Sep 27 18:34:28 dist-test-slave-19nl krb5kdc[8120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:28.813594 8227 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:28.813982 8227 webserver.cc:460] Webserver started at http://127.5.151.193:46431/ using document root <none> and password file <none>
I20260927 18:34:28.814200 8227 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:28.814265 8227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:28.814407 8227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:28.815454 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "94e33e97f0e849f0a5bc694bba315c13"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "60de0d812997694997486405f76a7d6e"
server_key_iv: "9dc4fbedbc49ecc82c9e597fb0297e5a"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.815804 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "94e33e97f0e849f0a5bc694bba315c13"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "60de0d812997694997486405f76a7d6e"
server_key_iv: "9dc4fbedbc49ecc82c9e597fb0297e5a"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.817082 8227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:28.817794 8242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:28.817963 8227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:28.818058 8227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "94e33e97f0e849f0a5bc694bba315c13"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "60de0d812997694997486405f76a7d6e"
server_key_iv: "9dc4fbedbc49ecc82c9e597fb0297e5a"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.818140 8227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:28.836191 8227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:28.837961 8227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:28.838107 8227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:28.838308 8227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:28.838636 8227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:28.838721 8227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:28.838762 8227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:28.838791 8227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:28.844434 8227 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:34915
I20260927 18:34:28.844476 8355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:34915 every 8 connection(s)
I20260927 18:34:28.844782 8227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:28.848856 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8227
I20260927 18:34:28.848927 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:28.849095 5727 external_mini_cluster.cc:1428] Setting key 4af427ab03bd4363bd624e2fdd405744
Sep 27 18:34:28 dist-test-slave-19nl krb5kdc[8120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:28.854301 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:28.845507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:37723 (local address 127.5.151.254:41453)
0927 18:34:28.845575 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:28.845577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:28.845748 (+ 171us) server_negotiation.cc:407] Connection header received
0927 18:34:28.846195 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:28.846196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:28.846205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:28.846259 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:28.847085 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:28.847574 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:28.848449 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:28.848629 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:28.848699 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:28.851213 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:28.851214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:28.851215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:28.851223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:28.852966 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:28.853349 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:28.853350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:28.853350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:28.853420 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:28.853652 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:28.853653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:28.853654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:28.853802 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:28.853890 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:28.854116 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 18:34:28.854202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:34:28.854336 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:28.845476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41453 (local address 127.5.151.193:37723)
0927 18:34:28.845621 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:28.845660 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:28.846115 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:28.846304 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:28.846307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:28.846835 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:28.847011 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:28.847015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:28.847648 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:28.847648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:28.848376 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:28.848378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:28.848708 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:28.848709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:28.848803 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:28.849283 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:28.849292 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:28.851134 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:28.853048 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:28.853054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:28.853056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:28.853262 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:28.853488 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:28.853488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:28.853488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:28.853585 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:28.853884 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:28.853889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:28.853991 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 18:34:28.854071 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:34:28.854177 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260927 18:34:28.854830 8356 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41453
I20260927 18:34:28.854940 8356 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:28.855136 8356 heartbeater.cc:511] Master 127.5.151.254:41453 requested a full tablet report, sending...
I20260927 18:34:28.855712 8165 ts_manager.cc:194] Registered new tserver with Master: 94e33e97f0e849f0a5bc694bba315c13 (127.5.151.193:34915)
I20260927 18:34:28.856626 8165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:37723
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:28.873600 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41453
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:28.950019 8361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:28.950194 8361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:28.950215 8361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:28.951478 8361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:28.951515 8361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:28.951542 8361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:28.952860 8361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:28.953019 8361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:28.953168 8361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:28.957800 8367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:28.957808 8366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:28.957813 8369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:28.957931 8361 server_base.cc:1034] running on GCE node
I20260927 18:34:28.958218 8361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:28.958474 8361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:28.959653 8361 hybrid_clock.cc:648] HybridClock initialized: now 1790534068959640 us; error 24 us; skew 500 ppm
Sep 27 18:34:28 dist-test-slave-19nl krb5kdc[8120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:28.961779 8361 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:28.962208 8361 webserver.cc:460] Webserver started at http://127.5.151.194:45723/ using document root <none> and password file <none>
I20260927 18:34:28.962409 8361 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:28.962473 8361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:28.962630 8361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:28.963532 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "dc54634f4ae347c0ae28c94129cbd340"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "089b0d1c505c888839fe51627116b285"
server_key_iv: "10b9dfec97c435882189f9c569b437a2"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.963866 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "dc54634f4ae347c0ae28c94129cbd340"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "089b0d1c505c888839fe51627116b285"
server_key_iv: "10b9dfec97c435882189f9c569b437a2"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.965055 8361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:34:28.965703 8376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:28.965860 8361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:28.965950 8361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "dc54634f4ae347c0ae28c94129cbd340"
format_stamp: "Formatted at 2026-09-27 18:34:28 on dist-test-slave-19nl"
server_key: "089b0d1c505c888839fe51627116b285"
server_key_iv: "10b9dfec97c435882189f9c569b437a2"
server_key_version: "encryptionkey@0"
I20260927 18:34:28.966025 8361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:29.004694 8361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:29.006464 8361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:29.006650 8361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:29.006879 8361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:29.007184 8361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:29.007236 8361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:29.007268 8361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:29.007293 8361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:29.012728 8361 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:45913
I20260927 18:34:29.012776 8489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:45913 every 8 connection(s)
I20260927 18:34:29.013067 8361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:29.018083 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8361
I20260927 18:34:29.018189 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:29.018409 5727 external_mini_cluster.cc:1428] Setting key 22b127367a76a2a213d47b485b3c98af
Sep 27 18:34:29 dist-test-slave-19nl krb5kdc[8120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:29.022215 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:29.013740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:36299 (local address 127.5.151.254:41453)
0927 18:34:29.013801 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:29.013804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:29.014015 (+ 211us) server_negotiation.cc:407] Connection header received
0927 18:34:29.014440 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:29.014441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:29.014451 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:29.014501 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:29.015366 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.015839 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.016720 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.016903 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.016957 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:29.019174 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:29.019175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:29.019176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:29.019184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:29.020873 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:29.021264 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:29.021265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:29.021265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:29.021327 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:29.021594 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:29.021595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:29.021595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:29.021725 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:29.021784 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:29.022040 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 18:34:29.022129 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:34:29.022392 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:29.013721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41453 (local address 127.5.151.194:36299)
0927 18:34:29.013904 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:29.013937 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:29.014372 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:29.014569 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:29.014572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:29.015087 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:29.015261 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:29.015265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.015917 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.015918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:29.016650 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:29.016651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.016974 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.016975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:29.017068 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:29.017479 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:29.017487 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:29.019087 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:29.020963 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:29.020969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:29.020971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:29.021178 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:29.021392 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:29.021393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:29.021393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:29.021531 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:29.021776 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:29.021781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:29.021915 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 18:34:29.022021 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:34:29.022164 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260927 18:34:29.023066 8490 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41453
I20260927 18:34:29.023144 8490 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:29.023321 8490 heartbeater.cc:511] Master 127.5.151.254:41453 requested a full tablet report, sending...
I20260927 18:34:29.023703 8165 ts_manager.cc:194] Registered new tserver with Master: dc54634f4ae347c0ae28c94129cbd340 (127.5.151.194:45913)
I20260927 18:34:29.024259 8165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:36299
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:29.041143 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41453
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:29.118593 8495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:29.118785 8495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:29.118808 8495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:29.119911 8495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:29.119946 8495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:29.119969 8495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:29.121379 8495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:29.121546 8495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:29.121701 8495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:29.126430 8501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:29.126535 8495 server_base.cc:1034] running on GCE node
W20260927 18:34:29.126436 8503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:29.126443 8500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:29.126806 8495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:29.127054 8495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:29.128214 8495 hybrid_clock.cc:648] HybridClock initialized: now 1790534069128205 us; error 38 us; skew 500 ppm
Sep 27 18:34:29 dist-test-slave-19nl krb5kdc[8120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534069, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:29.130509 8495 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:29.130970 8495 webserver.cc:460] Webserver started at http://127.5.151.195:45825/ using document root <none> and password file <none>
I20260927 18:34:29.131175 8495 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:29.131237 8495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:29.131378 8495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:29.132310 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "0239bdb3b91f402da7cdb3626ece651f"
format_stamp: "Formatted at 2026-09-27 18:34:29 on dist-test-slave-19nl"
server_key: "ce8d52586fd61397def49b501600f966"
server_key_iv: "816b471a2e14f633b62974473ad7d573"
server_key_version: "encryptionkey@0"
I20260927 18:34:29.132648 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "0239bdb3b91f402da7cdb3626ece651f"
format_stamp: "Formatted at 2026-09-27 18:34:29 on dist-test-slave-19nl"
server_key: "ce8d52586fd61397def49b501600f966"
server_key_iv: "816b471a2e14f633b62974473ad7d573"
server_key_version: "encryptionkey@0"
I20260927 18:34:29.133883 8495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:29.134536 8510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:29.134718 8495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:29.134809 8495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "0239bdb3b91f402da7cdb3626ece651f"
format_stamp: "Formatted at 2026-09-27 18:34:29 on dist-test-slave-19nl"
server_key: "ce8d52586fd61397def49b501600f966"
server_key_iv: "816b471a2e14f633b62974473ad7d573"
server_key_version: "encryptionkey@0"
I20260927 18:34:29.134887 8495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:29.147078 8495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:29.148765 8495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:29.148913 8495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:29.149109 8495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:29.149398 8495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:29.149456 8495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:29.149495 8495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:29.149523 8495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:29.155488 8495 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:44041
I20260927 18:34:29.155558 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:44041 every 8 connection(s)
I20260927 18:34:29.155824 8495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:29 dist-test-slave-19nl krb5kdc[8120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534069, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:29.164062 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:29.156456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:52879 (local address 127.5.151.254:41453)
0927 18:34:29.156495 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:29.156497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:29.156766 (+ 269us) server_negotiation.cc:407] Connection header received
0927 18:34:29.157239 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:29.157240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:29.157251 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:29.157314 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:29.158150 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.158704 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.159623 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.159829 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.159914 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:29.161756 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:29.161758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:29.161758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:29.161766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:29.162992 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:29.163352 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:29.163352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:29.163353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:29.163404 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:29.163594 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:29.163595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:29.163595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:29.163684 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:29.163730 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:29.163946 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 18:34:29.164002 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:34:29.164207 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:29.156463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41453 (local address 127.5.151.195:52879)
0927 18:34:29.156647 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:29.156687 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:29.157145 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:29.157383 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:29.157387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:29.157892 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:29.158062 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:29.158066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.158788 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.158790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:29.159538 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:29.159539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:29.159923 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:29.159925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:29.160036 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:29.160464 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:29.160472 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:29.161670 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:29.163063 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:29.163068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:29.163069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:29.163272 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:29.163455 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:29.163456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:29.163456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:29.163545 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:29.163739 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:29.163743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:29.163846 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 18:34:29.163923 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:34:29.164027 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260927 18:34:29.164768 8624 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41453
I20260927 18:34:29.164855 8624 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:29.164978 8624 heartbeater.cc:511] Master 127.5.151.254:41453 requested a full tablet report, sending...
I20260927 18:34:29.165313 8165 ts_manager.cc:194] Registered new tserver with Master: 0239bdb3b91f402da7cdb3626ece651f (127.5.151.195:44041)
I20260927 18:34:29.165570 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8495
I20260927 18:34:29.165649 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:29.165802 5727 external_mini_cluster.cc:1428] Setting key e4a7787245fc39bdf4deb17a3c2ad34c
I20260927 18:34:29.165817 8165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:52879
I20260927 18:34:29.166632 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:34:29.177388 8204 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:29.176507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.5.151.254:41453)
0927 18:34:29.176576 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:29.176578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:29.176680 (+ 102us) server_negotiation.cc:407] Connection header received
0927 18:34:29.176759 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:29.176760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:29.176770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:29.176839 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:29.177325 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42972: BlockingRecv error: recv got EOF from 127.0.0.1:42972 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:34:29.178193 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8227
I20260927 18:34:29.182164 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8361
I20260927 18:34:29.186061 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8495
I20260927 18:34:29.189782 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8133
2026-09-27T18:34:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1779 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:29 dist-test-slave-19nl krb5kdc[8636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:29 dist-test-slave-19nl krb5kdc[8636](info): set up 2 sockets
Sep 27 18:34:29 dist-test-slave-19nl krb5kdc[8636](info): commencing operation
krb5kdc: starting...
W20260927 18:34:30.178104 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:30 dist-test-slave-19nl krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:30.282677 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43931
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:43931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:30.359184 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:30.359444 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:30.359467 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:30.360514 8649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:30.360548 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:30.360560 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:30.360572 8649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:30.360584 8649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:30.361907 8649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:43931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:30.362089 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:30.362246 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:30.366780 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:30.366809 8649 server_base.cc:1034] running on GCE node
W20260927 18:34:30.366781 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:30.366779 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:30.367194 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:30.367488 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:30.368624 8649 hybrid_clock.cc:648] HybridClock initialized: now 1790534070368614 us; error 29 us; skew 500 ppm
Sep 27 18:34:30 dist-test-slave-19nl krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534070, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:30.370904 8649 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:30.371302 8649 webserver.cc:460] Webserver started at http://127.5.151.254:35821/ using document root <none> and password file <none>
I20260927 18:34:30.371516 8649 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:30.371559 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:30.371704 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:30.372671 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "1fc194dcbffc44de85169cf794966ba2"
format_stamp: "Formatted at 2026-09-27 18:34:30 on dist-test-slave-19nl"
server_key: "50dfe5b8c28e2b8337b82fd382c70d15"
server_key_iv: "07488e5460b817d340bdc5b10aa92d71"
server_key_version: "encryptionkey@0"
I20260927 18:34:30.373021 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "1fc194dcbffc44de85169cf794966ba2"
format_stamp: "Formatted at 2026-09-27 18:34:30 on dist-test-slave-19nl"
server_key: "50dfe5b8c28e2b8337b82fd382c70d15"
server_key_iv: "07488e5460b817d340bdc5b10aa92d71"
server_key_version: "encryptionkey@0"
I20260927 18:34:30.374223 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 18:34:30.374957 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:30.375133 8649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:30.375247 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "1fc194dcbffc44de85169cf794966ba2"
format_stamp: "Formatted at 2026-09-27 18:34:30 on dist-test-slave-19nl"
server_key: "50dfe5b8c28e2b8337b82fd382c70d15"
server_key_iv: "07488e5460b817d340bdc5b10aa92d71"
server_key_version: "encryptionkey@0"
I20260927 18:34:30.375315 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:30.393575 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:30.395375 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:30.395526 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:30.399353 8649 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:43931
I20260927 18:34:30.399399 8716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:43931 every 8 connection(s)
I20260927 18:34:30.399735 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:30.400524 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:30.402684 8717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2: Bootstrap starting.
I20260927 18:34:30.403262 8717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:30.403554 8717 log.cc:824] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:30.404176 8717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2: No bootstrap required, opened a new log
I20260927 18:34:30.405610 8717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc194dcbffc44de85169cf794966ba2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43931 } }
I20260927 18:34:30.405737 8717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:30.405794 8717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc194dcbffc44de85169cf794966ba2, State: Initialized, Role: FOLLOWER
I20260927 18:34:30.405926 8717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc194dcbffc44de85169cf794966ba2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43931 } }
I20260927 18:34:30.406001 8717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:30.406057 8717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:30.406101 8717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:30.406680 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8649
I20260927 18:34:30.406749 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:30.406916 5727 external_mini_cluster.cc:1428] Setting key 7af5cf92e8a401a91d9205f9a8ed273f
I20260927 18:34:30.406998 8717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc194dcbffc44de85169cf794966ba2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43931 } }
I20260927 18:34:30.407117 8717 leader_election.cc:302] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fc194dcbffc44de85169cf794966ba2; no voters:
I20260927 18:34:30.407275 8717 leader_election.cc:288] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:30.407305 8720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:30.407502 8720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [term 1 LEADER]: Becoming Leader. State: Replica: 1fc194dcbffc44de85169cf794966ba2, State: Running, Role: LEADER
I20260927 18:34:30.407603 8717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:30.407622 8720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc194dcbffc44de85169cf794966ba2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43931 } }
I20260927 18:34:30.408198 8724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fc194dcbffc44de85169cf794966ba2. Latest consensus state: current_term: 1 leader_uuid: "1fc194dcbffc44de85169cf794966ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc194dcbffc44de85169cf794966ba2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43931 } } }
I20260927 18:34:30.408301 8724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:30.408203 8723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fc194dcbffc44de85169cf794966ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc194dcbffc44de85169cf794966ba2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43931 } } }
I20260927 18:34:30.408411 8723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:30.408619 8732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:30.409169 8732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:30.410826 8732 catalog_manager.cc:1347] Generated new cluster ID: fe767e38e4634d9982c7f597fd3d83a5
I20260927 18:34:30.410884 8732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:30 dist-test-slave-19nl krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:30.418927 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:30.407299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.5.151.254:43931)
0927 18:34:30.407811 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:30.407820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:30.407852 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:34:30.407971 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:30.407976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:30.408018 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:30.409525 (+ 1507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:30.409639 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:30.410780 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:30.411885 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:30.412466 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:30.412558 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:30.415168 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:30.415172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:30.415173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:30.415183 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:30.417387 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:30.417759 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:30.417763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:30.417763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:30.417871 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:30.418064 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:30.418065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:30.418065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:30.418352 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:30.418508 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:30.418603 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 18:34:30.418734 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":76,"threads_started":1}
I20260927 18:34:30.419868 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:34:30.428534 8732 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:30.429210 8732 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:30.431167 8732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fc194dcbffc44de85169cf794966ba2: Generated new TSK 0
I20260927 18:34:30.431334 8732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:34:30.438004 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:30.434637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.5.151.254:43931)
0927 18:34:30.434689 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:30.434691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:30.434790 (+ 99us) server_negotiation.cc:407] Connection header received
0927 18:34:30.434843 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:30.434844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:30.434851 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:30.434897 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:30.435285 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:30.435733 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:30.436780 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:30.436978 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:30.437035 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:30.437612 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:30.437613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:30.437613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:30.437762 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:30.437775 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:30.437780 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:30.437824 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:30.437901 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 18:34:30.437919 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:34:30.440397 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8649
2026-09-27T18:34:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1248 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:30 dist-test-slave-19nl krb5kdc[8750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:30 dist-test-slave-19nl krb5kdc[8750](info): set up 2 sockets
Sep 27 18:34:30 dist-test-slave-19nl krb5kdc[8750](info): commencing operation
krb5kdc: starting...
W20260927 18:34:31.417970 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:31 dist-test-slave-19nl krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:31.522329 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39995
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:39995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:31.599916 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:31.600101 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:31.600121 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:31.601223 8763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:31.601265 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:31.601280 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:31.601326 8763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:31.601348 8763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:31.602833 8763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:39995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:31.603067 8763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:31.603284 8763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:31.607846 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:31.607868 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:31.608059 8763 server_base.cc:1034] running on GCE node
W20260927 18:34:31.607872 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:31.608337 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:31.608603 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:31.609747 8763 hybrid_clock.cc:648] HybridClock initialized: now 1790534071609736 us; error 27 us; skew 500 ppm
Sep 27 18:34:31 dist-test-slave-19nl krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534071, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:31.616187 8763 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:31.616636 8763 webserver.cc:460] Webserver started at http://127.5.151.254:39483/ using document root <none> and password file <none>
I20260927 18:34:31.616875 8763 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:31.616942 8763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:31.617096 8763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:31.618117 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "b7d77b0a3b3742f3ac8d318845c3085c"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "e8ac40274bfa5a91a715dbffc438c888"
server_key_iv: "9cacd658bd6294d934b6c10b7dc6e341"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.618482 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "b7d77b0a3b3742f3ac8d318845c3085c"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "e8ac40274bfa5a91a715dbffc438c888"
server_key_iv: "9cacd658bd6294d934b6c10b7dc6e341"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.619747 8763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:31.620440 8778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:31.620605 8763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:31.620692 8763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "b7d77b0a3b3742f3ac8d318845c3085c"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "e8ac40274bfa5a91a715dbffc438c888"
server_key_iv: "9cacd658bd6294d934b6c10b7dc6e341"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.620774 8763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:31.631975 8763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:31.633577 8763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:31.633728 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:31.637621 8763 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:39995
I20260927 18:34:31.637671 8830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:39995 every 8 connection(s)
I20260927 18:34:31.638005 8763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:31.638985 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:31.641120 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c: Bootstrap starting.
I20260927 18:34:31.641616 8831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:31.641835 8831 log.cc:824] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:31.642429 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c: No bootstrap required, opened a new log
I20260927 18:34:31.643985 8831 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39995 } }
I20260927 18:34:31.644117 8831 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:31.644140 8831 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d77b0a3b3742f3ac8d318845c3085c, State: Initialized, Role: FOLLOWER
I20260927 18:34:31.644236 8831 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39995 } }
I20260927 18:34:31.644284 8831 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:31.644304 8831 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:31.644328 8831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:31.645035 8831 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39995 } }
I20260927 18:34:31.645133 8831 leader_election.cc:302] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7d77b0a3b3742f3ac8d318845c3085c; no voters:
I20260927 18:34:31.645249 8831 leader_election.cc:288] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:31.645324 8834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:31.645478 8834 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [term 1 LEADER]: Becoming Leader. State: Replica: b7d77b0a3b3742f3ac8d318845c3085c, State: Running, Role: LEADER
I20260927 18:34:31.645526 8831 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:31.645565 8834 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39995 } }
I20260927 18:34:31.646015 8836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7d77b0a3b3742f3ac8d318845c3085c. Latest consensus state: current_term: 1 leader_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39995 } } }
I20260927 18:34:31.645954 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7d77b0a3b3742f3ac8d318845c3085c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39995 } } }
I20260927 18:34:31.646164 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:31.646350 8836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:31.646607 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8763
I20260927 18:34:31.646608 8842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:31.646703 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:31.646876 5727 external_mini_cluster.cc:1428] Setting key c2866a0d61d070bb8d3ff1d5ee12e2a2
I20260927 18:34:31.647119 8842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:31.648635 8842 catalog_manager.cc:1347] Generated new cluster ID: 1b972d5d4c00482c9a8e7abd50c8075c
I20260927 18:34:31.648687 8842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:31 dist-test-slave-19nl krb5kdc[8750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:31.658574 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:31.647516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.5.151.254:39995)
0927 18:34:31.647895 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:31.647903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:31.647931 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:34:31.648035 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:31.648041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:31.648116 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:31.649059 (+ 943us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:31.649141 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.650591 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:31.651610 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.652180 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:31.652271 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:31.655037 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:31.655041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:31.655041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:31.655053 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:31.657010 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:31.657369 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:31.657372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:31.657373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:31.657477 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:31.657693 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:31.657694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:31.657694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:31.657972 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:31.658126 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:31.658195 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:34:31.658344 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":74,"threads_started":1}
I20260927 18:34:31.670056 8842 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:31.670782 8842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:31.684940 8842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7d77b0a3b3742f3ac8d318845c3085c: Generated new TSK 0
I20260927 18:34:31.685146 8842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:31.715042 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39995
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:31.808480 8857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:31.808662 8857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:31.808684 8857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:31.809716 8857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:31.809757 8857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:31.809780 8857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:31.811017 8857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:31.811172 8857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:31.811321 8857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:31.816144 8857 server_base.cc:1034] running on GCE node
W20260927 18:34:31.816061 8863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:31.816071 8862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:31.816061 8865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:31.816443 8857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:31.816746 8857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:31.817891 8857 hybrid_clock.cc:648] HybridClock initialized: now 1790534071817876 us; error 31 us; skew 500 ppm
Sep 27 18:34:31 dist-test-slave-19nl krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534071, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:31.820287 8857 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:31.820731 8857 webserver.cc:460] Webserver started at http://127.5.151.193:34109/ using document root <none> and password file <none>
I20260927 18:34:31.820955 8857 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:31.820995 8857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:31.821127 8857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:31.822042 8857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "394da7690ca340769f8f10ef39c6d22b"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "2c0e1202d3613aae3efae9e016438491"
server_key_iv: "37d89a2bd3acca7202c0d83331dafe4c"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.822351 8857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "394da7690ca340769f8f10ef39c6d22b"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "2c0e1202d3613aae3efae9e016438491"
server_key_iv: "37d89a2bd3acca7202c0d83331dafe4c"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.823699 8857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:31.824434 8872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:31.824627 8857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:31.824695 8857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "394da7690ca340769f8f10ef39c6d22b"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "2c0e1202d3613aae3efae9e016438491"
server_key_iv: "37d89a2bd3acca7202c0d83331dafe4c"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.824780 8857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:31.842310 8857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:31.843946 8857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:31.844094 8857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:31.844301 8857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:31.844585 8857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:31.844615 8857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:31.844635 8857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:31.844681 8857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:31.850764 8857 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:34473
I20260927 18:34:31.850807 8985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:34473 every 8 connection(s)
I20260927 18:34:31.851094 8857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:34:31 dist-test-slave-19nl krb5kdc[8750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534071, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:31.859042 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:31.851792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:53301 (local address 127.5.151.254:39995)
0927 18:34:31.851870 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:31.851873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:31.852059 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:34:31.852512 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:31.852513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:31.852524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:31.852598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:31.853421 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.853873 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:31.854738 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.854912 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:31.854960 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:31.856840 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:31.856842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:31.856842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:31.856848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:31.858052 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:31.858343 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:31.858344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:31.858344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:31.858388 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:31.858519 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:31.858520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:31.858520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:31.858657 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:31.858702 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:31.858921 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 18:34:31.858983 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:34:31.859187 8988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:31.851753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39995 (local address 127.5.151.193:53301)
0927 18:34:31.851936 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:31.851977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:31.852435 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:31.852654 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:31.852658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:31.853189 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:31.853355 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:31.853359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.853942 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:31.853943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:31.854674 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:31.854676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.854977 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:31.854978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:31.855062 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:31.855470 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:31.855477 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:31.856748 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:31.858112 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:31.858116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:31.858118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:31.858288 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:31.858422 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:31.858422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:31.858423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:31.858490 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:31.858706 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:31.858711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:31.858804 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 18:34:31.858892 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:34:31.858995 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260927 18:34:31.859632 8986 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39995
I20260927 18:34:31.859723 8986 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:31.859809 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8857
I20260927 18:34:31.859920 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:31.859916 8986 heartbeater.cc:511] Master 127.5.151.254:39995 requested a full tablet report, sending...
I20260927 18:34:31.860146 5727 external_mini_cluster.cc:1428] Setting key 06243828f94b108414d0c3ca3c69aebb
I20260927 18:34:31.860479 8795 ts_manager.cc:194] Registered new tserver with Master: 394da7690ca340769f8f10ef39c6d22b (127.5.151.193:34473)
I20260927 18:34:31.861176 8795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:53301
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:31.883082 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39995
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:31.957746 8991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:31.957942 8991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:31.957978 8991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:31.959190 8991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:31.959249 8991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:31.959291 8991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:31.960623 8991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:31.960866 8991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:31.961069 8991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:31.965547 8999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:31.965538 8997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:31.965538 8996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:31.965786 8991 server_base.cc:1034] running on GCE node
I20260927 18:34:31.965983 8991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:31.966202 8991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:31.967347 8991 hybrid_clock.cc:648] HybridClock initialized: now 1790534071967340 us; error 36 us; skew 500 ppm
Sep 27 18:34:31 dist-test-slave-19nl krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534071, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:31.969460 8991 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:31.969887 8991 webserver.cc:460] Webserver started at http://127.5.151.194:40987/ using document root <none> and password file <none>
I20260927 18:34:31.970080 8991 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:31.970140 8991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:31.970258 8991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:31.971258 8991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "7bd1d12b25a044f0856812e4e6eaac8c"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "ee6bcde2e0efdec0b7e9a00433af0ff0"
server_key_iv: "b77568241be5bedbc9c86f2bb5097a33"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.971583 8991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "7bd1d12b25a044f0856812e4e6eaac8c"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "ee6bcde2e0efdec0b7e9a00433af0ff0"
server_key_iv: "b77568241be5bedbc9c86f2bb5097a33"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.972779 8991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:34:31.973421 9006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:31.973575 8991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:31.973670 8991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "7bd1d12b25a044f0856812e4e6eaac8c"
format_stamp: "Formatted at 2026-09-27 18:34:31 on dist-test-slave-19nl"
server_key: "ee6bcde2e0efdec0b7e9a00433af0ff0"
server_key_iv: "b77568241be5bedbc9c86f2bb5097a33"
server_key_version: "encryptionkey@0"
I20260927 18:34:31.973747 8991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:31.986640 8991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:31.988988 8991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:31.989146 8991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:31.989367 8991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:31.989717 8991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:31.989763 8991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:31.989797 8991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:31.989825 8991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:31.995610 8991 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:43817
I20260927 18:34:31.995673 9119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:43817 every 8 connection(s)
I20260927 18:34:31.995940 8991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:31.997185 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 8991
I20260927 18:34:31.997308 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:31.997493 5727 external_mini_cluster.cc:1428] Setting key c441e7c8cac5f4ea9dc38a2e198525da
Sep 27 18:34:31 dist-test-slave-19nl krb5kdc[8750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534071, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:32.005497 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:31.996618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:57531 (local address 127.5.151.254:39995)
0927 18:34:31.996675 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:31.996677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:31.996805 (+ 128us) server_negotiation.cc:407] Connection header received
0927 18:34:31.997263 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:31.997265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:31.997280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:31.997351 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:31.998355 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.999128 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.000092 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.000409 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.000481 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:32.002760 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:32.002762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:32.002762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:32.002770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:32.004111 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:32.004503 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:32.004504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:32.004505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:32.004567 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:32.004876 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:32.004877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:32.004877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:32.004978 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:32.005030 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:32.005353 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 18:34:32.005423 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:34:32.005762 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:31.996546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39995 (local address 127.5.151.194:57531)
0927 18:34:31.996697 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:31.996734 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:31.997166 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:31.997436 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:31.997440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:31.998112 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:31.998286 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:31.998290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:31.999227 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:31.999228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:32.000009 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:32.000011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.000497 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.000498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:32.000590 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:32.001056 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:32.001065 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:32.002668 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:32.004208 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:32.004214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:32.004216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:32.004428 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:32.004654 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:32.004655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:32.004656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:32.004794 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:32.005044 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:32.005049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:32.005208 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 18:34:32.005326 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:34:32.005501 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
I20260927 18:34:32.006204 9120 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39995
I20260927 18:34:32.006313 9120 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:32.006487 9120 heartbeater.cc:511] Master 127.5.151.254:39995 requested a full tablet report, sending...
I20260927 18:34:32.006897 8795 ts_manager.cc:194] Registered new tserver with Master: 7bd1d12b25a044f0856812e4e6eaac8c (127.5.151.194:43817)
I20260927 18:34:32.007256 8795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:57531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:32.021013 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39995
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:32.097397 9125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:32.097604 9125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:32.097640 9125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:32.098778 9125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:32.098832 9125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:32.098875 9125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:32.100212 9125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:32.100447 9125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:32.100648 9125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:32.105243 9130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:32.105245 9125 server_base.cc:1034] running on GCE node
W20260927 18:34:32.105295 9131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:32.105404 9133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:32.105592 9125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:32.105831 9125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:32.106984 9125 hybrid_clock.cc:648] HybridClock initialized: now 1790534072106975 us; error 37 us; skew 500 ppm
Sep 27 18:34:32 dist-test-slave-19nl krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:32.109213 9125 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:32.109609 9125 webserver.cc:460] Webserver started at http://127.5.151.195:43387/ using document root <none> and password file <none>
I20260927 18:34:32.109776 9125 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:32.109813 9125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:32.109894 9125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:32.110836 9125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "b9555f811323430a9693c4d4300d5b76"
format_stamp: "Formatted at 2026-09-27 18:34:32 on dist-test-slave-19nl"
server_key: "b19bc64252380011b0058ebe5698d224"
server_key_iv: "5a4f0d49b1305733192cdcdd678af662"
server_key_version: "encryptionkey@0"
I20260927 18:34:32.111107 9125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "b9555f811323430a9693c4d4300d5b76"
format_stamp: "Formatted at 2026-09-27 18:34:32 on dist-test-slave-19nl"
server_key: "b19bc64252380011b0058ebe5698d224"
server_key_iv: "5a4f0d49b1305733192cdcdd678af662"
server_key_version: "encryptionkey@0"
I20260927 18:34:32.112200 9125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:32.112785 9140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:32.112938 9125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:32.113010 9125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "b9555f811323430a9693c4d4300d5b76"
format_stamp: "Formatted at 2026-09-27 18:34:32 on dist-test-slave-19nl"
server_key: "b19bc64252380011b0058ebe5698d224"
server_key_iv: "5a4f0d49b1305733192cdcdd678af662"
server_key_version: "encryptionkey@0"
I20260927 18:34:32.113056 9125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:32.119515 9125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:32.121235 9125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:32.121330 9125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:32.121476 9125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:32.121690 9125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:32.121718 9125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:32.121737 9125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:32.121750 9125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:32.127100 9125 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:43649
I20260927 18:34:32.127167 9253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:43649 every 8 connection(s)
I20260927 18:34:32.127485 9125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:32 dist-test-slave-19nl krb5kdc[8750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:32.134757 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 9125
I20260927 18:34:32.134855 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:32.135002 5727 external_mini_cluster.cc:1428] Setting key 9bb1ec6878122a3b9a2fa4947cb2f80e
I20260927 18:34:32.135953 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:32.128098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:42393 (local address 127.5.151.254:39995)
0927 18:34:32.128137 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:32.128140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:32.128305 (+ 165us) server_negotiation.cc:407] Connection header received
0927 18:34:32.128775 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:32.128776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:32.128789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:32.128839 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:32.129748 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.130235 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.131296 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.131495 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.131577 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:32.133388 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:32.133390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:32.133390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:32.133397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:32.134499 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:32.134955 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:32.134956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:32.134956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:32.135013 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:32.135272 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:32.135273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:32.135274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:32.135415 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:32.135482 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:32.135785 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:34:32.135874 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:34:32.136099 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:32.128058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39995 (local address 127.5.151.195:42393)
0927 18:34:32.128187 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:32.128232 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:32.128676 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:32.128898 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:32.128902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:32.129414 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:32.129639 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:32.129652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.130324 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.130325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:32.131181 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:32.131184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.131586 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.131587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:32.131705 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:32.132063 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:32.132070 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:32.133284 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:32.134619 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:32.134626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:32.134628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:32.134884 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:32.135079 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:32.135079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:32.135080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:32.135198 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:32.135482 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:32.135488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:32.135630 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 18:34:32.135758 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:34:32.135903 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260927 18:34:32.136539 9254 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39995
I20260927 18:34:32.136610 9254 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:32.136725 9254 heartbeater.cc:511] Master 127.5.151.254:39995 requested a full tablet report, sending...
I20260927 18:34:32.136987 8795 ts_manager.cc:194] Registered new tserver with Master: b9555f811323430a9693c4d4300d5b76 (127.5.151.195:43649)
I20260927 18:34:32.137339 8795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:42393
I20260927 18:34:32.146761 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:34:32 dist-test-slave-19nl krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 18:34:32 dist-test-slave-19nl krb5kdc[8750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:32.167378 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:32.160871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.5.151.254:39995)
0927 18:34:32.160948 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:32.160949 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:32.160987 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:34:32.161029 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:32.161030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:32.161038 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:32.161085 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:32.161593 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.162038 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.162794 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:32.162972 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:32.163055 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:32.165079 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:32.165080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:32.165080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:32.165086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:32.166411 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:32.166757 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:32.166758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:32.166758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:32.166805 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:32.166957 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:32.166958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:32.166958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:32.167074 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:32.167156 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:32.167230 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 18:34:32.167282 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 18:34:32.167634 8795 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48918
I20260927 18:34:32.168073 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8857
I20260927 18:34:32.173195 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8991
I20260927 18:34:32.177044 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 9125
I20260927 18:34:32.180760 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 8763
2026-09-27T18:34:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1742 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:32 dist-test-slave-19nl krb5kdc[9266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:32 dist-test-slave-19nl krb5kdc[9266](info): set up 2 sockets
Sep 27 18:34:32 dist-test-slave-19nl krb5kdc[9266](info): commencing operation
krb5kdc: starting...
W20260927 18:34:33.150218 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:33.257543 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36125
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:33.333364 9279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:33.333557 9279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:33.333578 9279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:33.334730 9279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:33.334769 9279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:33.334782 9279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:33.334833 9279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:33.334859 9279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:33.336315 9279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:36125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:33.336551 9279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:33.336772 9279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:33.341502 9279 server_base.cc:1034] running on GCE node
W20260927 18:34:33.341478 9285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:33.341460 9284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:33.341475 9287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:33.341967 9279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:33.342259 9279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:33.343427 9279 hybrid_clock.cc:648] HybridClock initialized: now 1790534073343419 us; error 35 us; skew 500 ppm
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:33.345795 9279 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:33.346195 9279 webserver.cc:460] Webserver started at http://127.5.151.254:37001/ using document root <none> and password file <none>
I20260927 18:34:33.346403 9279 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:33.346467 9279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:33.346635 9279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:33.347648 9279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "d24b5af932d4453596a48c0438b475f2"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "c7ab2efdbe2a9b7e7e5cee3418e0672f"
server_key_iv: "55f8640b357a56b9ae6d2891dd9d30d9"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.348004 9279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "d24b5af932d4453596a48c0438b475f2"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "c7ab2efdbe2a9b7e7e5cee3418e0672f"
server_key_iv: "55f8640b357a56b9ae6d2891dd9d30d9"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.349299 9279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:33.350008 9294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.350167 9279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:33.350263 9279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "d24b5af932d4453596a48c0438b475f2"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "c7ab2efdbe2a9b7e7e5cee3418e0672f"
server_key_iv: "55f8640b357a56b9ae6d2891dd9d30d9"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.350348 9279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:33.364020 9279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:33.365926 9279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:33.366084 9279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:33.369848 9279 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:36125
I20260927 18:34:33.369895 9346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:36125 every 8 connection(s)
I20260927 18:34:33.370390 9279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:33.371114 9347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:33.371580 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 9279
I20260927 18:34:33.371704 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:33.371881 5727 external_mini_cluster.cc:1428] Setting key ed8104d79400b1545476c41e32ca4d05
I20260927 18:34:33.373696 9347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2: Bootstrap starting.
I20260927 18:34:33.374253 9347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:33.374529 9347 log.cc:824] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:33.375213 9347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2: No bootstrap required, opened a new log
I20260927 18:34:33.377090 9347 raft_consensus.cc:348] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b5af932d4453596a48c0438b475f2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36125 } }
I20260927 18:34:33.377254 9347 raft_consensus.cc:374] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:33.377303 9347 raft_consensus.cc:729] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24b5af932d4453596a48c0438b475f2, State: Initialized, Role: FOLLOWER
I20260927 18:34:33.377429 9347 consensus_queue.cc:260] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b5af932d4453596a48c0438b475f2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36125 } }
I20260927 18:34:33.377521 9347 raft_consensus.cc:388] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:33.377561 9347 raft_consensus.cc:482] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:33.377606 9347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:33.378073 9347 raft_consensus.cc:504] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b5af932d4453596a48c0438b475f2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36125 } }
I20260927 18:34:33.378180 9347 leader_election.cc:302] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d24b5af932d4453596a48c0438b475f2; no voters:
I20260927 18:34:33.378343 9347 leader_election.cc:288] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:33.378376 9352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:33.378490 9352 raft_consensus.cc:686] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [term 1 LEADER]: Becoming Leader. State: Replica: d24b5af932d4453596a48c0438b475f2, State: Running, Role: LEADER
I20260927 18:34:33.378619 9347 sys_catalog.cc:564] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:33.378641 9352 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b5af932d4453596a48c0438b475f2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36125 } }
I20260927 18:34:33.378947 9352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24b5af932d4453596a48c0438b475f2. Latest consensus state: current_term: 1 leader_uuid: "d24b5af932d4453596a48c0438b475f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b5af932d4453596a48c0438b475f2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36125 } } }
I20260927 18:34:33.378963 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d24b5af932d4453596a48c0438b475f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b5af932d4453596a48c0438b475f2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36125 } } }
I20260927 18:34:33.379119 9352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:33.379125 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:33.379411 9360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:33.379948 9360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:33.382496 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.372371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.5.151.254:36125)
0927 18:34:33.372548 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:33.372563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:33.372585 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:34:33.372980 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:33.372985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:33.373025 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:33.373177 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:33.373841 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.374874 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.375702 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.376108 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.376240 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.378662 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:33.378666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:33.378666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:33.378677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:33.381105 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.381458 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.381461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.381461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.381542 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.381738 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.381739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.381739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.381953 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:33.382100 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:33.382192 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 18:34:33.382353 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260927 18:34:33.383769 9360 catalog_manager.cc:1347] Generated new cluster ID: 8c84ed049a784b5d985cfcfe37d52ba7
I20260927 18:34:33.383818 9360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:33.411432 9360 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:33.412150 9360 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:33.417999 9360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d24b5af932d4453596a48c0438b475f2: Generated new TSK 0
I20260927 18:34:33.418160 9360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:33.443602 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36125
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:33.521183 9373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:33.521374 9373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:33.521397 9373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:33.522617 9373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:33.522655 9373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:33.522678 9373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:33.523977 9373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:33.524137 9373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:33.524295 9373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:33.529066 9373 server_base.cc:1034] running on GCE node
W20260927 18:34:33.529034 9379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:33.529034 9381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:33.529036 9378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:33.529531 9373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:33.529946 9373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:33.531098 9373 hybrid_clock.cc:648] HybridClock initialized: now 1790534073531089 us; error 28 us; skew 500 ppm
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:33.533556 9373 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:33.533967 9373 webserver.cc:460] Webserver started at http://127.5.151.193:38227/ using document root <none> and password file <none>
I20260927 18:34:33.534191 9373 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:33.534236 9373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:33.534412 9373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:33.535485 9373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "1d94856d47f04e4e90792a47d6593c54"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "459bec5a9731082f8d7369e8dbf23a21"
server_key_iv: "aac6a0b773ce1429701e2c4182e9a51c"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.535840 9373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "1d94856d47f04e4e90792a47d6593c54"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "459bec5a9731082f8d7369e8dbf23a21"
server_key_iv: "aac6a0b773ce1429701e2c4182e9a51c"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.537081 9373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:33.537804 9388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.537997 9373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:33.538086 9373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "1d94856d47f04e4e90792a47d6593c54"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "459bec5a9731082f8d7369e8dbf23a21"
server_key_iv: "aac6a0b773ce1429701e2c4182e9a51c"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.538139 9373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:33.550936 9373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:33.552948 9373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:33.553103 9373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:33.553282 9373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:33.553560 9373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:33.553611 9373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.553653 9373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:33.553679 9373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.559132 9373 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:42915
I20260927 18:34:33.559192 9501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:42915 every 8 connection(s)
I20260927 18:34:33.559469 9373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:33.567508 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.560128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:50197 (local address 127.5.151.254:36125)
0927 18:34:33.560198 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:33.560200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:33.560356 (+ 156us) server_negotiation.cc:407] Connection header received
0927 18:34:33.561190 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:33.561190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:33.561199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:33.561260 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:33.561767 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.562236 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.563108 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.563309 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.563360 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.565271 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:33.565273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:33.565273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:33.565279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:33.566436 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.566827 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.566828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.566828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.566886 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.567072 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.567073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.567073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.567159 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:33.567204 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:33.567398 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 18:34:33.567452 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:34:33.567687 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.560078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36125 (local address 127.5.151.193:50197)
0927 18:34:33.560237 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:33.560274 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:33.561110 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:33.561313 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:33.561316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:33.561534 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:33.561699 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:33.561702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.562322 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.562322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:33.563039 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:33.563040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.563377 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.563378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:33.563465 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.563930 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:33.563939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:33.565188 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:33.566503 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:33.566509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:33.566510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:33.566754 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:33.566940 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:33.566940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:33.566941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:33.567025 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:33.567205 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:33.567209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:33.567303 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 18:34:33.567374 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:34:33.567475 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260927 18:34:33.568037 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 9373
I20260927 18:34:33.568142 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:33.568244 9502 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36125
I20260927 18:34:33.568328 9502 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:33.568346 5727 external_mini_cluster.cc:1428] Setting key 6fb1c670bd1b2205a75943c2f1d8100b
I20260927 18:34:33.568535 9502 heartbeater.cc:511] Master 127.5.151.254:36125 requested a full tablet report, sending...
I20260927 18:34:33.569080 9310 ts_manager.cc:194] Registered new tserver with Master: 1d94856d47f04e4e90792a47d6593c54 (127.5.151.193:42915)
I20260927 18:34:33.569947 9310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:50197
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:33.591326 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36125
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:33.668085 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:33.668267 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:33.668287 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:33.669327 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:33.669358 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:33.669382 9507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:33.670806 9507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:33.670979 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:33.671131 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:33.675555 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:33.675560 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:33.675657 9507 server_base.cc:1034] running on GCE node
W20260927 18:34:33.675560 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:33.675958 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:33.676196 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:33.677352 9507 hybrid_clock.cc:648] HybridClock initialized: now 1790534073677350 us; error 37 us; skew 500 ppm
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:33.680145 9507 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:33.680553 9507 webserver.cc:460] Webserver started at http://127.5.151.194:38509/ using document root <none> and password file <none>
I20260927 18:34:33.680742 9507 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:33.680780 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:33.680861 9507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:33.681756 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "3db068081ffe408b945d53f24bf8e83c"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "94ba691205dfcde5984a4f574db77279"
server_key_iv: "cd1fb6dc85b6533b6f140dac9cfe9930"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.682034 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "3db068081ffe408b945d53f24bf8e83c"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "94ba691205dfcde5984a4f574db77279"
server_key_iv: "cd1fb6dc85b6533b6f140dac9cfe9930"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.683262 9507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 18:34:33.683881 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.684023 9507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.684100 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "3db068081ffe408b945d53f24bf8e83c"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "94ba691205dfcde5984a4f574db77279"
server_key_iv: "cd1fb6dc85b6533b6f140dac9cfe9930"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.684150 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:33.695508 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:33.697124 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:33.697229 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:33.697383 9507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:33.697594 9507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:33.697621 9507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.697640 9507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:33.697652 9507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.703101 9507 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:45591
I20260927 18:34:33.703173 9635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:45591 every 8 connection(s)
I20260927 18:34:33.703457 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:33.705361 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 9507
I20260927 18:34:33.705476 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:33.705650 5727 external_mini_cluster.cc:1428] Setting key be9043382ff5e7cfb260657d679d5853
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:33.713580 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.704158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:46729 (local address 127.5.151.254:36125)
0927 18:34:33.704219 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:33.704221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:33.704379 (+ 158us) server_negotiation.cc:407] Connection header received
0927 18:34:33.705342 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:33.705343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:33.705354 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:33.705431 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:33.706120 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.706813 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.708264 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.708578 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.708651 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.710922 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:33.710924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:33.710924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:33.710930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:33.712224 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.712668 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.712668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.712669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.712713 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.712972 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.712972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.712972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.713060 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:33.713107 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:33.713448 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 18:34:33.713512 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:34:33.713841 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.704094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36125 (local address 127.5.151.194:46729)
0927 18:34:33.704256 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:33.704295 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:33.705212 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:33.705499 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:33.705504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:33.705858 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:33.706041 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:33.706046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.706900 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.706901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:33.708185 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:33.708188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.708657 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.708658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:33.708758 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.709218 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:33.709226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:33.710836 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:33.712303 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:33.712310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:33.712312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:33.712593 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:33.712773 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:33.712774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:33.712774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:33.712908 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:33.713120 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:33.713125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:33.713303 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 18:34:33.713434 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:34:33.713600 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260927 18:34:33.714304 9636 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36125
I20260927 18:34:33.714404 9636 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
I20260927 18:34:33.714612 9636 heartbeater.cc:511] Master 127.5.151.254:36125 requested a full tablet report, sending...
I20260927 18:34:33.715029 9310 ts_manager.cc:194] Registered new tserver with Master: 3db068081ffe408b945d53f24bf8e83c (127.5.151.194:45591)
I20260927 18:34:33.715400 9310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:46729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:33.729247 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36125
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:33.807997 9641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:33.808177 9641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:33.808199 9641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:33.809295 9641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:33.809333 9641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:33.809382 9641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:33.810823 9641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:33.811046 9641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:33.811259 9641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:33.815994 9647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:33.815985 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:33.816001 9649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:33.816071 9641 server_base.cc:1034] running on GCE node
I20260927 18:34:33.816380 9641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:33.816617 9641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:33.817771 9641 hybrid_clock.cc:648] HybridClock initialized: now 1790534073817764 us; error 35 us; skew 500 ppm
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:33.820006 9641 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:33.820402 9641 webserver.cc:460] Webserver started at http://127.5.151.195:39257/ using document root <none> and password file <none>
I20260927 18:34:33.820614 9641 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:33.820679 9641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:33.820842 9641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:33.821846 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "84a0b20737ef416dbad5195e3940a89c"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "0bf13dcf2ba03bef17c313325e1d830e"
server_key_iv: "7ccbe75bbea404d759fcb9724bcb996e"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.822211 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "84a0b20737ef416dbad5195e3940a89c"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "0bf13dcf2ba03bef17c313325e1d830e"
server_key_iv: "7ccbe75bbea404d759fcb9724bcb996e"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.823608 9641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:33.824293 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.824453 9641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:33.824545 9641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "84a0b20737ef416dbad5195e3940a89c"
format_stamp: "Formatted at 2026-09-27 18:34:33 on dist-test-slave-19nl"
server_key: "0bf13dcf2ba03bef17c313325e1d830e"
server_key_iv: "7ccbe75bbea404d759fcb9724bcb996e"
server_key_version: "encryptionkey@0"
I20260927 18:34:33.824622 9641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:33.836508 9641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:33.838096 9641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:33.838240 9641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:33.838443 9641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:33.838752 9641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:33.838814 9641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.838852 9641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:33.838877 9641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:33.844609 9641 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:33521
I20260927 18:34:33.844642 9769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:33521 every 8 connection(s)
I20260927 18:34:33.844937 9641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:33.853129 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.845579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:54431 (local address 127.5.151.254:36125)
0927 18:34:33.845661 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:33.845663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:33.845855 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:34:33.846775 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:33.846776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:33.846785 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:33.846838 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:33.847341 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.847803 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.848670 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.848836 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.848883 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.850909 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:33.850911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:33.850911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:33.850916 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:33.851998 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.852327 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.852327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.852328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.852375 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.852560 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.852561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.852561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.852650 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:33.852696 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:33.853005 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 18:34:33.853071 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:34:33.853259 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.845566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36125 (local address 127.5.151.195:54431)
0927 18:34:33.845723 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:33.845762 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:33.846687 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:33.846888 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:33.846892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:33.847106 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:33.847277 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:33.847281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.847871 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.847872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:33.848605 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:33.848607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.848901 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.848902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:33.848990 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.849391 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:33.849398 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:33.850830 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:33.852062 (+ 1232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:33.852067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:33.852069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:33.852265 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:33.852427 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:33.852428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:33.852428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:33.852511 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:33.852706 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:33.852713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:33.852839 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 18:34:33.852954 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:34:33.853059 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260927 18:34:33.853319 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 9641
I20260927 18:34:33.853413 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:33.853597 5727 external_mini_cluster.cc:1428] Setting key 21db17e5018a11c53de939187437a924
I20260927 18:34:33.853824 9770 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36125
I20260927 18:34:33.853919 9770 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:33.854091 9770 heartbeater.cc:511] Master 127.5.151.254:36125 requested a full tablet report, sending...
I20260927 18:34:33.854463 9310 ts_manager.cc:194] Registered new tserver with Master: 84a0b20737ef416dbad5195e3940a89c (127.5.151.195:33521)
I20260927 18:34:33.854846 9310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:54431
I20260927 18:34:33.865319 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:33.887318 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.880180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.5.151.193:42915)
0927 18:34:33.880404 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:33.880410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:33.880423 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:34:33.880451 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:33.880453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:33.880474 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:33.880538 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:33.881044 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.881750 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.882407 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.882754 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.882815 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.884801 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:33.884808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:33.884808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:33.884814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:33.886118 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.886430 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.886432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.886433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.886511 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.886830 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.886831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.886831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.886959 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:33.887104 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:33.887140 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 18:34:33.887201 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:33.894454 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:33.888507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.5.151.254:36125)
0927 18:34:33.888553 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:33.888555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:33.888583 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:34:33.888640 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:33.888641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:33.888649 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:33.888691 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:33.889117 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.889576 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.890196 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:33.890376 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:33.890421 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:33.892416 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:33.892417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:33.892417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:33.892423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:33.893515 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.893801 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.893802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.893802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.893846 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:33.894023 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:33.894024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:33.894024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:33.894150 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:33.894222 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:33.894307 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 18:34:33.894360 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 18:34:33.894708 9310 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57494
I20260927 18:34:33.895068 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 9373
I20260927 18:34:33.899591 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 9507
I20260927 18:34:33.903901 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 9641
I20260927 18:34:33.908001 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 9279
2026-09-27T18:34:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1727 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9782](info): set up 2 sockets
Sep 27 18:34:33 dist-test-slave-19nl krb5kdc[9782](info): commencing operation
krb5kdc: starting...
W20260927 18:34:34.906255 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:34 dist-test-slave-19nl krb5kdc[9782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:35.015581 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:37867
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:37867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:35.091763 9795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:35.091966 9795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:35.092005 9795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:35.093137 9795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:35.093197 9795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:35.093221 9795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:35.093250 9795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:35.093271 9795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:35.094782 9795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:37867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:37867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:35.095038 9795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:35.095248 9795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:35.099823 9800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:35.099829 9801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:35.100031 9795 server_base.cc:1034] running on GCE node
W20260927 18:34:35.099840 9803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:35.100328 9795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:35.100602 9795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:35.101761 9795 hybrid_clock.cc:648] HybridClock initialized: now 1790534075101756 us; error 34 us; skew 500 ppm
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534075, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:35.104013 9795 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:35.104328 9795 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:35.104394 9795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:35.104524 9795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:35.105417 9795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "dfad815687154d8fae46ee380365272e"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "10ad669c4dba1dcf21f4b2e77f960c77"
server_key_iv: "62ae3d91f67e79564b04b157d78cf432"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.105724 9795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "dfad815687154d8fae46ee380365272e"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "10ad669c4dba1dcf21f4b2e77f960c77"
server_key_iv: "62ae3d91f67e79564b04b157d78cf432"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.106902 9795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:35.107586 9809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.107743 9795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:35.107846 9795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "dfad815687154d8fae46ee380365272e"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "10ad669c4dba1dcf21f4b2e77f960c77"
server_key_iv: "62ae3d91f67e79564b04b157d78cf432"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.107916 9795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:35.124444 9795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:35.126158 9795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:35.126302 9795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:35.130052 9795 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:37867
I20260927 18:34:35.130074 9861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:37867 every 8 connection(s)
I20260927 18:34:35.130332 9795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:35.131214 9862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:35.133309 9862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e: Bootstrap starting.
I20260927 18:34:35.133800 9862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:35.134021 9862 log.cc:824] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:35.134656 9862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e: No bootstrap required, opened a new log
I20260927 18:34:35.136040 9862 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfad815687154d8fae46ee380365272e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37867 } }
I20260927 18:34:35.136214 9862 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:35.136271 9862 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfad815687154d8fae46ee380365272e, State: Initialized, Role: FOLLOWER
I20260927 18:34:35.136392 9862 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfad815687154d8fae46ee380365272e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37867 } }
I20260927 18:34:35.136445 9862 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:35.136464 9862 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:35.136493 9862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:35.137241 9862 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfad815687154d8fae46ee380365272e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37867 } }
I20260927 18:34:35.137320 9862 leader_election.cc:302] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfad815687154d8fae46ee380365272e; no voters:
I20260927 18:34:35.137439 9862 leader_election.cc:288] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:35.137512 9865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:35.137669 9865 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [term 1 LEADER]: Becoming Leader. State: Replica: dfad815687154d8fae46ee380365272e, State: Running, Role: LEADER
I20260927 18:34:35.137706 9862 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:35.137744 9865 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfad815687154d8fae46ee380365272e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37867 } }
I20260927 18:34:35.138091 9867 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfad815687154d8fae46ee380365272e. Latest consensus state: current_term: 1 leader_uuid: "dfad815687154d8fae46ee380365272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfad815687154d8fae46ee380365272e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37867 } } }
I20260927 18:34:35.138185 9867 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:35.138091 9866 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfad815687154d8fae46ee380365272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfad815687154d8fae46ee380365272e" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37867 } } }
I20260927 18:34:35.138355 9866 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:35.138662 9873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:35.139156 9873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:35.139428 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 9795
I20260927 18:34:35.139526 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:35.139667 5727 external_mini_cluster.cc:1428] Setting key 3a874cb6679037e50bde98cd55bc265d
I20260927 18:34:35.140610 9873 catalog_manager.cc:1347] Generated new cluster ID: 8bbfbd99c77544398b1e476a7cc92079
I20260927 18:34:35.140661 9873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:35.148543 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.140049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.5.151.254:37867)
0927 18:34:35.140205 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.140213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.140245 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:34:35.140366 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.140371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.140410 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.140529 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:35.141175 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.142053 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.142871 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.143298 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.143419 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.145484 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:35.145488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:35.145488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:35.145499 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:35.147098 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.147444 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.147447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.147447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.147531 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.147730 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.147731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.147731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.147951 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:35.148131 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.148221 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.148369 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260927 18:34:35.154752 9873 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:35.155418 9873 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:35.159054 9873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e: Generated new TSK 0
I20260927 18:34:35.159196 9873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:35.189749 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:37867
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:35.291064 9888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:35.291270 9888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:35.291303 9888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:35.292371 9888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:35.292428 9888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:35.292490 9888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:35.293949 9888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:37867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:35.294196 9888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:35.294399 9888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:35.299160 9896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:35.299163 9894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:35.299194 9888 server_base.cc:1034] running on GCE node
W20260927 18:34:35.299167 9893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:35.299638 9888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:35.299844 9888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:35.300979 9888 hybrid_clock.cc:648] HybridClock initialized: now 1790534075300970 us; error 26 us; skew 500 ppm
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534075, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:35.303323 9888 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:35.303671 9888 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:35.303736 9888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:35.303876 9888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:35.304842 9888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "e1397bb3200b4890a49320421db43c62"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "8c07c9d713361bf2e0eaec718be3845f"
server_key_iv: "c37bc23556d91cce1c688315ce302835"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.305215 9888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "e1397bb3200b4890a49320421db43c62"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "8c07c9d713361bf2e0eaec718be3845f"
server_key_iv: "c37bc23556d91cce1c688315ce302835"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.306383 9888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:35.307085 9902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.307247 9888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.307349 9888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "e1397bb3200b4890a49320421db43c62"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "8c07c9d713361bf2e0eaec718be3845f"
server_key_iv: "c37bc23556d91cce1c688315ce302835"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.307428 9888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:35.328116 9888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:35.329831 9888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:35.329977 9888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:35.330199 9888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:35.330489 9888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:35.330536 9888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.330595 9888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:35.330622 9888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.336320 9888 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:33197
I20260927 18:34:35.336362 10015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:33197 every 8 connection(s)
I20260927 18:34:35.336648 9888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534075, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:35.344862 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 9888
I20260927 18:34:35.344921 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.337282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:46297 (local address 127.5.151.254:37867)
0927 18:34:35.337352 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.337354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.337493 (+ 139us) server_negotiation.cc:407] Connection header received
0927 18:34:35.338003 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.338004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.338018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.338064 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:35.339016 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.339587 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.340521 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.340712 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.340760 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.342706 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:35.342708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:35.342708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:35.342715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:35.343798 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.344130 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.344131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.344131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.344176 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.344363 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.344363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.344363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.344450 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:35.344497 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.344722 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.344803 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:34:35.344942 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.337241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:37867 (local address 127.5.151.193:46297)
0927 18:34:35.337396 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.337428 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.337857 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.338123 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.338126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.338419 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:35.338945 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.338950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.339659 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.339660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.340452 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.340454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.340780 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.340781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.340883 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.341281 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:35.341289 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:35.342630 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:35.343872 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:35.343879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:35.343880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:35.344072 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:35.344227 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:35.344227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:35.344227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:35.344312 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:35.344499 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:35.344503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:35.344599 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.344673 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.344772 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260927 18:34:35.344960 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:35.345126 5727 external_mini_cluster.cc:1428] Setting key a62de3fd391c31d8cac0c65ba1c9ae75
I20260927 18:34:35.345659 10016 heartbeater.cc:347] Connected to a master server at 127.5.151.254:37867
I20260927 18:34:35.345739 10016 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:35.345835 10016 heartbeater.cc:511] Master 127.5.151.254:37867 requested a full tablet report, sending...
I20260927 18:34:35.346288 9826 ts_manager.cc:194] Registered new tserver with Master: e1397bb3200b4890a49320421db43c62 (127.5.151.193:33197)
I20260927 18:34:35.346930 9826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:46297
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:35.367563 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:37867
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:35.445988 10021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:35.446163 10021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:35.446185 10021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:35.447252 10021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:35.447288 10021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:35.447312 10021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:35.448632 10021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:37867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:35.448828 10021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:35.449020 10021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:35.453748 10021 server_base.cc:1034] running on GCE node
W20260927 18:34:35.453755 10027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:35.453732 10026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:35.453730 10029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:35.454106 10021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:35.454344 10021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:35.455478 10021 hybrid_clock.cc:648] HybridClock initialized: now 1790534075455467 us; error 27 us; skew 500 ppm
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534075, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:35.457783 10021 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:35.458142 10021 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:35.458210 10021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:35.458338 10021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:35.459323 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "648b4cfaff284db3aaf05faa7b8e37c3"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "4ecc09d19adee8bcb1f1058859523844"
server_key_iv: "f8c38ab5c8bfe7c0dad1c103db5257c4"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.459683 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "648b4cfaff284db3aaf05faa7b8e37c3"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "4ecc09d19adee8bcb1f1058859523844"
server_key_iv: "f8c38ab5c8bfe7c0dad1c103db5257c4"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.460889 10021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:34:35.461536 10035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.461681 10021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:35.461776 10021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "648b4cfaff284db3aaf05faa7b8e37c3"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "4ecc09d19adee8bcb1f1058859523844"
server_key_iv: "f8c38ab5c8bfe7c0dad1c103db5257c4"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.461871 10021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:35.475441 10021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:35.477145 10021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:35.477299 10021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:35.477511 10021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:35.477805 10021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:35.477862 10021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.477905 10021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:35.477931 10021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.483853 10021 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:37313
I20260927 18:34:35.483893 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:37313 every 8 connection(s)
I20260927 18:34:35.484156 10021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534075, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:35.492168 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10021
I20260927 18:34:35.492250 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:35.492452 5727 external_mini_cluster.cc:1428] Setting key 64e623fbb0f4c2969bdb2fa27378126e
I20260927 18:34:35.494189 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.484932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:60515 (local address 127.5.151.254:37867)
0927 18:34:35.485005 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.485007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.485246 (+ 239us) server_negotiation.cc:407] Connection header received
0927 18:34:35.485894 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.485896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.485908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.485988 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:35.487087 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.487803 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.488740 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.488944 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.489020 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.491186 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:35.491187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:35.491187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:35.491194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:35.492767 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.493234 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.493236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.493236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.493294 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.493560 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.493561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.493561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.493675 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:35.493749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.494022 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.494102 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:34:35.494619 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.484906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:37867 (local address 127.5.151.194:60515)
0927 18:34:35.485120 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.485165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.485778 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.486058 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.486062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.486446 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:35.486997 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.487002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.487896 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.487897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.488660 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.488667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.489063 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.489064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.489182 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.489636 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:35.489644 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:35.491106 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:35.492882 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:35.492889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:35.492891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:35.493176 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:35.493376 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:35.493377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:35.493377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:35.493485 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:35.493741 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:35.493747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:35.493886 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.493990 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.494142 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260927 18:34:35.495076 10149 heartbeater.cc:347] Connected to a master server at 127.5.151.254:37867
I20260927 18:34:35.495177 10149 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:35.495349 10149 heartbeater.cc:511] Master 127.5.151.254:37867 requested a full tablet report, sending...
I20260927 18:34:35.495687 9826 ts_manager.cc:194] Registered new tserver with Master: 648b4cfaff284db3aaf05faa7b8e37c3 (127.5.151.194:37313)
I20260927 18:34:35.496038 9826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:60515
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:35.514856 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:37867
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:35.590401 10154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:35.590618 10154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:35.590641 10154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:35.591729 10154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:35.591771 10154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:35.591797 10154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:35.593196 10154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:37867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:35.593364 10154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:35.593533 10154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:35.598206 10160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:35.598245 10159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:35.598209 10162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:35.598435 10154 server_base.cc:1034] running on GCE node
I20260927 18:34:35.598709 10154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:35.598945 10154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:35.600101 10154 hybrid_clock.cc:648] HybridClock initialized: now 1790534075600092 us; error 38 us; skew 500 ppm
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534075, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:35.602464 10154 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:35.602866 10154 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:35.602942 10154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:35.603075 10154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:35.604084 10154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "7e353be113b547ec8fb80bbeabc421a9"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "e530667c2b46fecab820750f0740047b"
server_key_iv: "4881e4f1bf873d3d347b80788c370cb5"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.604435 10154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "7e353be113b547ec8fb80bbeabc421a9"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "e530667c2b46fecab820750f0740047b"
server_key_iv: "4881e4f1bf873d3d347b80788c370cb5"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.605624 10154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 18:34:35.606262 10168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.606428 10154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:35.606530 10154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "7e353be113b547ec8fb80bbeabc421a9"
format_stamp: "Formatted at 2026-09-27 18:34:35 on dist-test-slave-19nl"
server_key: "e530667c2b46fecab820750f0740047b"
server_key_iv: "4881e4f1bf873d3d347b80788c370cb5"
server_key_version: "encryptionkey@0"
I20260927 18:34:35.606614 10154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:35.633849 10154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:35.635502 10154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:35.635648 10154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:35.635842 10154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:35.636129 10154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:35.636181 10154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.636216 10154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:35.636243 10154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:35.642130 10154 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:43683
I20260927 18:34:35.642160 10281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:43683 every 8 connection(s)
I20260927 18:34:35.642405 10154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:35 dist-test-slave-19nl krb5kdc[9782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534075, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:35.649204 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10154
I20260927 18:34:35.649272 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:35.649446 5727 external_mini_cluster.cc:1428] Setting key cf1a4c56016cd4e0920a5f252d6a2e51
I20260927 18:34:35.650728 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.643100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:58211 (local address 127.5.151.254:37867)
0927 18:34:35.643156 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.643158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.643348 (+ 190us) server_negotiation.cc:407] Connection header received
0927 18:34:35.643782 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.643783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.643793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.643843 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:35.644637 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.645096 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.645970 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.646140 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.646185 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.648088 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:35.648089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:35.648089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:35.648095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:35.649370 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.649774 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.649775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.649776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.649839 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.650084 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.650085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.650086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.650202 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:35.650276 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.650543 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.650644 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:34:35.650874 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.643095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:37867 (local address 127.5.151.195:58211)
0927 18:34:35.643232 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.643262 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.643688 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.643893 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.643895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.644130 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:35.644569 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.644573 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.645166 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.645167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.645903 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.645904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.646203 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.646204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.646294 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.646781 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:35.646788 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:35.648013 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:35.649444 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:35.649451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:35.649453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:35.649694 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:35.649917 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:35.649918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:35.649918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:35.650018 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:35.650267 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:35.650273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:35.650404 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.650499 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.650657 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260927 18:34:35.651250 10282 heartbeater.cc:347] Connected to a master server at 127.5.151.254:37867
I20260927 18:34:35.651340 10282 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:35.651479 10282 heartbeater.cc:511] Master 127.5.151.254:37867 requested a full tablet report, sending...
I20260927 18:34:35.651829 9826 ts_manager.cc:194] Registered new tserver with Master: 7e353be113b547ec8fb80bbeabc421a9 (127.5.151.195:43683)
I20260927 18:34:35.652334 9826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:58211
I20260927 18:34:35.660879 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:34:35.667284 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.661991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.5.151.254:37867)
0927 18:34:35.662053 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.662059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.662105 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:34:35.662142 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.662143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.662151 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.662195 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:35.662720 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.663149 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.663824 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.663989 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.664034 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.665168 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:35.665169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:35.665169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:35.665175 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:35.666365 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.666697 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.666698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.666698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.666744 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:35.666917 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:35.666917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:35.666917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:35.667036 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:35.667104 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.667183 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.667232 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:34:35.668565 9826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:35.670007 9826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:35.675992 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.672191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.5.151.193:33197)
0927 18:34:35.672362 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.672367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.672478 (+ 111us) server_negotiation.cc:407] Connection header received
0927 18:34:35.672611 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.672613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.672629 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.672674 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:35.672911 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.673968 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.675081 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.675600 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.675688 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.675739 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.675876 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.675914 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260927 18:34:35.676024 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.672094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:33197 (local address 127.0.0.1:55538)
0927 18:34:35.672384 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.672420 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.672510 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.672795 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.672795 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.672796 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:35.672831 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.672832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.674081 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.674082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.674996 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.674998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.675692 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.675693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.675767 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.675778 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.675861 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.675909 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":25,"thread_start_us":44,"threads_started":1}
I20260927 18:34:35.676112 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.672263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:43683 (local address 127.0.0.1:59916)
0927 18:34:35.672590 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.672596 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.672647 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.672875 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.672876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.672877 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:35.672916 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.672917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.673737 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.673738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.674904 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.674907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.675763 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.675764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.675826 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.675830 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.675876 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.675919 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":70,"threads_started":1}
I20260927 18:34:35.676246 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.672117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:37313 (local address 127.0.0.1:35008)
0927 18:34:35.672469 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.672478 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.672532 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.672762 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.672765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.672767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:35.672829 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.672831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.674080 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.674081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.675168 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.675170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.676020 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.676020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.676090 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.676093 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.676150 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.676192 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":22,"thread_start_us":66,"threads_started":1}
I20260927 18:34:35.676260 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.672213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.5.151.194:37313)
0927 18:34:35.672412 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.672416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.672535 (+ 119us) server_negotiation.cc:407] Connection header received
0927 18:34:35.672590 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.672592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.672607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.672673 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:35.672903 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.673963 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.675244 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.675893 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.675944 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.675991 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.676167 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.676210 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":103,"threads_started":1}
I20260927 18:34:35.676590 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.672338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.5.151.195:43683)
0927 18:34:35.672467 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.672470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.672651 (+ 181us) server_negotiation.cc:407] Connection header received
0927 18:34:35.672708 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.672710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.672722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.672807 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:35.673002 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.673642 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.674984 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.675616 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.675677 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.675731 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.676020 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.676541 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260927 18:34:35.676895 10083 tablet_service.cc:1467] Processing CreateTablet for tablet 178b4714154749a98d07e49195d20d28 (DEFAULT_TABLE table=test-table [id=e96d3aced99f4644b94fa9fb7c8ba584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:35.676920 9950 tablet_service.cc:1467] Processing CreateTablet for tablet 178b4714154749a98d07e49195d20d28 (DEFAULT_TABLE table=test-table [id=e96d3aced99f4644b94fa9fb7c8ba584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:35.677258 10083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178b4714154749a98d07e49195d20d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:35.677258 9950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178b4714154749a98d07e49195d20d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:35.677207 10216 tablet_service.cc:1467] Processing CreateTablet for tablet 178b4714154749a98d07e49195d20d28 (DEFAULT_TABLE table=test-table [id=e96d3aced99f4644b94fa9fb7c8ba584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:35.677461 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178b4714154749a98d07e49195d20d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:35.679788 10299 tablet_bootstrap.cc:492] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Bootstrap starting.
I20260927 18:34:35.680454 10299 tablet_bootstrap.cc:654] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:35.680482 10300 tablet_bootstrap.cc:492] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62: Bootstrap starting.
I20260927 18:34:35.681047 10299 log.cc:824] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:35.681097 10300 tablet_bootstrap.cc:654] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:35.681291 10301 tablet_bootstrap.cc:492] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Bootstrap starting.
I20260927 18:34:35.681658 10300 log.cc:824] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:35.681799 10299 tablet_bootstrap.cc:492] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: No bootstrap required, opened a new log
I20260927 18:34:35.681861 10299 ts_tablet_manager.cc:1397] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:35.681947 10301 tablet_bootstrap.cc:654] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:35.682348 10301 log.cc:824] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:35.683012 10301 tablet_bootstrap.cc:492] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: No bootstrap required, opened a new log
I20260927 18:34:35.683012 10300 tablet_bootstrap.cc:492] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62: No bootstrap required, opened a new log
I20260927 18:34:35.683079 10300 ts_tablet_manager.cc:1397] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:34:35.683079 10301 ts_tablet_manager.cc:1397] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:34:35.683468 10299 raft_consensus.cc:348] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.683609 10299 raft_consensus.cc:374] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:35.683647 10299 raft_consensus.cc:729] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e353be113b547ec8fb80bbeabc421a9, State: Initialized, Role: FOLLOWER
I20260927 18:34:35.683751 10299 consensus_queue.cc:260] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.684022 10299 ts_tablet_manager.cc:1428] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:35.684139 10282 heartbeater.cc:503] Master 127.5.151.254:37867 was elected leader, sending a full tablet report...
I20260927 18:34:35.684923 10300 raft_consensus.cc:348] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.684923 10301 raft_consensus.cc:348] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.685070 10301 raft_consensus.cc:374] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:35.685070 10300 raft_consensus.cc:374] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:35.685104 10301 raft_consensus.cc:729] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648b4cfaff284db3aaf05faa7b8e37c3, State: Initialized, Role: FOLLOWER
I20260927 18:34:35.685104 10300 raft_consensus.cc:729] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1397bb3200b4890a49320421db43c62, State: Initialized, Role: FOLLOWER
I20260927 18:34:35.685206 10301 consensus_queue.cc:260] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.685206 10300 consensus_queue.cc:260] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.685463 10300 ts_tablet_manager.cc:1428] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:34:35.685463 10301 ts_tablet_manager.cc:1428] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:34:35.685485 10016 heartbeater.cc:503] Master 127.5.151.254:37867 was elected leader, sending a full tablet report...
I20260927 18:34:35.685761 10149 heartbeater.cc:503] Master 127.5.151.254:37867 was elected leader, sending a full tablet report...
W20260927 18:34:35.734874 10150 tablet.cc:2369] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:35.837443 10017 tablet.cc:2369] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:35.862555 10305 raft_consensus.cc:482] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:35.862725 10305 raft_consensus.cc:504] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.863112 10305 leader_election.cc:288] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1397bb3200b4890a49320421db43c62 (127.5.151.193:33197), 648b4cfaff284db3aaf05faa7b8e37c3 (127.5.151.194:37313)
I20260927 18:34:35.865840 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.863364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:36349 (local address 127.5.151.193:33197)
0927 18:34:35.863414 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.863416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.863550 (+ 134us) server_negotiation.cc:407] Connection header received
0927 18:34:35.863663 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.863664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.863674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.863726 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:35.863863 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.864273 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.865212 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.865525 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.865575 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.865601 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.865734 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.865776 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 18:34:35.865898 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.863383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:38075 (local address 127.5.151.194:37313)
0927 18:34:35.863444 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.863446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.863454 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:35.863481 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.863481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.863488 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.863547 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:35.863737 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.864228 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.865192 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.865532 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.865611 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.865618 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.865788 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.865825 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:34:35.865844 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.863281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:33197 (local address 127.5.151.195:36349)
0927 18:34:35.863478 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.863493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.863563 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.863738 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.863739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.863741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:35.863800 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.863802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.864347 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.864347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.865116 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.865118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.865604 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.865604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.865674 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.865677 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.865715 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.865759 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":45,"thread_start_us":104,"threads_started":1}
I20260927 18:34:35.865855 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.863272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:37313 (local address 127.5.151.195:38075)
0927 18:34:35.863324 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.863335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:35.863375 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:35.863560 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:35.863561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:35.863563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:35.863646 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.863648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.864297 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.864298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.865108 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:35.865110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.865621 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.865622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:35.865696 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.865698 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:35.865764 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:34:35.865795 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":36}
I20260927 18:34:35.866324 9970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178b4714154749a98d07e49195d20d28" candidate_uuid: "7e353be113b547ec8fb80bbeabc421a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1397bb3200b4890a49320421db43c62" is_pre_election: true
I20260927 18:34:35.866360 10103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178b4714154749a98d07e49195d20d28" candidate_uuid: "7e353be113b547ec8fb80bbeabc421a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" is_pre_election: true
I20260927 18:34:35.866468 9970 raft_consensus.cc:2454] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e353be113b547ec8fb80bbeabc421a9 in term 0.
I20260927 18:34:35.866494 10103 raft_consensus.cc:2454] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e353be113b547ec8fb80bbeabc421a9 in term 0.
I20260927 18:34:35.866698 10170 leader_election.cc:302] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e353be113b547ec8fb80bbeabc421a9, e1397bb3200b4890a49320421db43c62; no voters:
I20260927 18:34:35.866840 10305 raft_consensus.cc:2781] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:35.866883 10305 raft_consensus.cc:482] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:35.866897 10305 raft_consensus.cc:3037] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:35.867501 10305 raft_consensus.cc:504] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.867624 10305 leader_election.cc:288] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [CANDIDATE]: Term 1 election: Requested vote from peers e1397bb3200b4890a49320421db43c62 (127.5.151.193:33197), 648b4cfaff284db3aaf05faa7b8e37c3 (127.5.151.194:37313)
I20260927 18:34:35.867739 9970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178b4714154749a98d07e49195d20d28" candidate_uuid: "7e353be113b547ec8fb80bbeabc421a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1397bb3200b4890a49320421db43c62"
I20260927 18:34:35.867766 10103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178b4714154749a98d07e49195d20d28" candidate_uuid: "7e353be113b547ec8fb80bbeabc421a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648b4cfaff284db3aaf05faa7b8e37c3"
I20260927 18:34:35.867813 9970 raft_consensus.cc:3037] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:35.867823 10103 raft_consensus.cc:3037] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:35.868657 10103 raft_consensus.cc:2454] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e353be113b547ec8fb80bbeabc421a9 in term 1.
I20260927 18:34:35.868657 9970 raft_consensus.cc:2454] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e353be113b547ec8fb80bbeabc421a9 in term 1.
I20260927 18:34:35.868829 10170 leader_election.cc:302] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e353be113b547ec8fb80bbeabc421a9, e1397bb3200b4890a49320421db43c62; no voters:
I20260927 18:34:35.868950 10305 raft_consensus.cc:2781] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:35.869110 10305 raft_consensus.cc:686] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 1 LEADER]: Becoming Leader. State: Replica: 7e353be113b547ec8fb80bbeabc421a9, State: Running, Role: LEADER
I20260927 18:34:35.869212 10305 consensus_queue.cc:237] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } }
I20260927 18:34:35.869942 9825 catalog_manager.cc:5495] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e353be113b547ec8fb80bbeabc421a9 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "7e353be113b547ec8fb80bbeabc421a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e353be113b547ec8fb80bbeabc421a9" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43683 } health_report { overall_health: HEALTHY } } }
W20260927 18:34:35.892920 10283 tablet.cc:2369] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:35.903184 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.901161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.5.151.195:43683)
0927 18:34:35.901213 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:35.901214 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:35.901222 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:35.901258 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:35.901259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:35.901274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:35.901333 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:35.901471 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.901899 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.902581 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:35.902743 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:35.902789 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:35.902925 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:35.903009 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:35.903064 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:35.903097 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 18:34:35.903112 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:34:35.904728 10103 raft_consensus.cc:1261] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 1 FOLLOWER]: Refusing update from remote peer 7e353be113b547ec8fb80bbeabc421a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:35.904737 9970 raft_consensus.cc:1261] T 178b4714154749a98d07e49195d20d28 P e1397bb3200b4890a49320421db43c62 [term 1 FOLLOWER]: Refusing update from remote peer 7e353be113b547ec8fb80bbeabc421a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:35.905033 10309 consensus_queue.cc:1035] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1397bb3200b4890a49320421db43c62" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:35.905153 10305 consensus_queue.cc:1035] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648b4cfaff284db3aaf05faa7b8e37c3" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:35.907250 10312 mvcc.cc:205] Tried to move back new op lower bound from 7334027574903447552 to 7334027574760779776. Current Snapshot: MvccSnapshot[applied={T|T < 7334027574903447552}]
I20260927 18:34:35.907433 10314 mvcc.cc:205] Tried to move back new op lower bound from 7334027574903447552 to 7334027574760779776. Current Snapshot: MvccSnapshot[applied={T|T < 7334027574903447552}]
I20260927 18:34:35.907565 10313 mvcc.cc:205] Tried to move back new op lower bound from 7334027574903447552 to 7334027574760779776. Current Snapshot: MvccSnapshot[applied={T|T < 7334027574903447552}]
I20260927 18:34:35.909323 9826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:35.909407 9826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:35.910157 9826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfad815687154d8fae46ee380365272e: Sending DeleteTablet for 3 replicas of tablet 178b4714154749a98d07e49195d20d28
I20260927 18:34:35.910519 10083 tablet_service.cc:1514] Processing DeleteTablet for tablet 178b4714154749a98d07e49195d20d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:35 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:35008
I20260927 18:34:35.910530 9950 tablet_service.cc:1514] Processing DeleteTablet for tablet 178b4714154749a98d07e49195d20d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:35 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:55538
I20260927 18:34:35.910629 10216 tablet_service.cc:1514] Processing DeleteTablet for tablet 178b4714154749a98d07e49195d20d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:35 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:59916
I20260927 18:34:35.910725 10321 tablet_replica.cc:330] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: stopping tablet replica
I20260927 18:34:35.910894 10323 tablet_replica.cc:330] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: stopping tablet replica
I20260927 18:34:35.911002 10323 raft_consensus.cc:2229] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:35.911087 10323 raft_consensus.cc:2258] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:35.911082 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 9888
I20260927 18:34:35.913059 10321 raft_consensus.cc:2229] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:35.913182 10321 raft_consensus.cc:2258] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:35.913586 10321 ts_tablet_manager.cc:1905] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:35.916281 10321 ts_tablet_manager.cc:1918] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:35.916352 10321 log.cc:1196] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/178b4714154749a98d07e49195d20d28
W20260927 18:34:35.916523 9811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:33197 (error 108)
I20260927 18:34:35.916630 10321 ts_tablet_manager.cc:1939] T 178b4714154749a98d07e49195d20d28 P 648b4cfaff284db3aaf05faa7b8e37c3: Deleting consensus metadata
I20260927 18:34:35.916886 10323 ts_tablet_manager.cc:1905] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:35.917016 9812 catalog_manager.cc:4841] TS 648b4cfaff284db3aaf05faa7b8e37c3 (127.5.151.194:37313): tablet 178b4714154749a98d07e49195d20d28 (table test-table [id=e96d3aced99f4644b94fa9fb7c8ba584]) successfully deleted
I20260927 18:34:35.917241 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:35.917131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:33197 (local address 127.0.0.1:55540)
0927 18:34:35.917171 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:35.917213 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 18:34:35.917362 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10021
W20260927 18:34:35.917351 9811 catalog_manager.cc:4584] TS e1397bb3200b4890a49320421db43c62 (127.5.151.193:33197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 178b4714154749a98d07e49195d20d28: Network error: Client connection negotiation failed: client connection to 127.5.151.193:33197: connect: Connection refused (error 111)
I20260927 18:34:35.918677 10323 ts_tablet_manager.cc:1918] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:35.918752 10323 log.cc:1196] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/178b4714154749a98d07e49195d20d28
I20260927 18:34:35.918987 10323 ts_tablet_manager.cc:1939] T 178b4714154749a98d07e49195d20d28 P 7e353be113b547ec8fb80bbeabc421a9: Deleting consensus metadata
I20260927 18:34:35.919327 9811 catalog_manager.cc:4841] TS 7e353be113b547ec8fb80bbeabc421a9 (127.5.151.195:43683): tablet 178b4714154749a98d07e49195d20d28 (table test-table [id=e96d3aced99f4644b94fa9fb7c8ba584]) successfully deleted
I20260927 18:34:35.923602 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10154
I20260927 18:34:35.929068 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 9795
2026-09-27T18:34:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2021 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T18:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:35Z Disabled control of system clock
I20260927 18:34:35.955754 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38029
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:38029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:34:36.040273 10328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:36.040459 10328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:36.040479 10328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:36.041560 10328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:36.041600 10328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:36.041612 10328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:36.041625 10328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:36.041674 10328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:36.043066 10328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:38029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:36.043303 10328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:36.043581 10328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:36.048270 10333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.048282 10336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.048285 10334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:36.048484 10328 server_base.cc:1034] running on GCE node
I20260927 18:34:36.048754 10328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:36.049077 10328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:36.050244 10328 hybrid_clock.cc:648] HybridClock initialized: now 1790534076050237 us; error 38 us; skew 500 ppm
I20260927 18:34:36.051378 10328 webserver.cc:460] Webserver started at http://127.5.151.254:32781/ using document root <none> and password file <none>
I20260927 18:34:36.051554 10328 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:36.051594 10328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:36.051677 10328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:36.052631 10328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "61375dd5d1064397929120ed35aa7a28"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "e401bc775e16d2dbfc8fd9b408d9edc9"
server_key_iv: "483c5385665fa488232e5844d3de3ff6"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.052924 10328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "61375dd5d1064397929120ed35aa7a28"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "e401bc775e16d2dbfc8fd9b408d9edc9"
server_key_iv: "483c5385665fa488232e5844d3de3ff6"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.054075 10328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:34:36.054764 10342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.054946 10328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:36.055032 10328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "61375dd5d1064397929120ed35aa7a28"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "e401bc775e16d2dbfc8fd9b408d9edc9"
server_key_iv: "483c5385665fa488232e5844d3de3ff6"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.055104 10328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:36.058123 10328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:36.058372 10328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:36.058504 10328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:36.062003 10328 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:38029
I20260927 18:34:36.062057 10394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:38029 every 8 connection(s)
I20260927 18:34:36.062336 10328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:36.062932 10395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:36.065284 10395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28: Bootstrap starting.
I20260927 18:34:36.065796 10395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:36.066076 10395 log.cc:824] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:36.066763 10395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28: No bootstrap required, opened a new log
I20260927 18:34:36.068183 10395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61375dd5d1064397929120ed35aa7a28" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38029 } }
I20260927 18:34:36.068297 10395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:36.068341 10395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61375dd5d1064397929120ed35aa7a28, State: Initialized, Role: FOLLOWER
I20260927 18:34:36.068472 10395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61375dd5d1064397929120ed35aa7a28" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38029 } }
I20260927 18:34:36.068552 10395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:36.068599 10395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:36.068643 10395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:36.069128 10395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61375dd5d1064397929120ed35aa7a28" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38029 } }
I20260927 18:34:36.069228 10395 leader_election.cc:302] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61375dd5d1064397929120ed35aa7a28; no voters:
I20260927 18:34:36.069406 10395 leader_election.cc:288] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:36.069442 10398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:36.069548 10398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [term 1 LEADER]: Becoming Leader. State: Replica: 61375dd5d1064397929120ed35aa7a28, State: Running, Role: LEADER
I20260927 18:34:36.069572 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10328
I20260927 18:34:36.069638 10395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:36.069612 10398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61375dd5d1064397929120ed35aa7a28" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38029 } }
I20260927 18:34:36.069670 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:36.069861 5727 external_mini_cluster.cc:1428] Setting key ce2b965d743cf8f1d6a5f39e22f3c7e3
I20260927 18:34:36.069940 10400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61375dd5d1064397929120ed35aa7a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61375dd5d1064397929120ed35aa7a28" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38029 } } }
I20260927 18:34:36.070016 10400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:36.070250 10399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61375dd5d1064397929120ed35aa7a28. Latest consensus state: current_term: 1 leader_uuid: "61375dd5d1064397929120ed35aa7a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61375dd5d1064397929120ed35aa7a28" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38029 } } }
I20260927 18:34:36.070381 10399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:36.070806 10402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:36.071406 10402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:36.072183 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.070423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.5.151.254:38029)
0927 18:34:36.070655 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.070664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.070694 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:34:36.070805 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.070811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.070871 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.071027 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.071765 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.071769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.071769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.071774 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.071824 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.071836 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.071919 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.071960 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.072016 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260927 18:34:36.073019 10402 catalog_manager.cc:1347] Generated new cluster ID: 6740353d1ce74b548cbae9edaa157d22
I20260927 18:34:36.073076 10402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:36.095043 10402 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:36.095199 10402 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:36.100288 10402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28: Generated new TSK 0
I20260927 18:34:36.100466 10402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:34:36.104275 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38029
--builtin_ntp_servers=127.5.151.212:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:34:36.216987 10419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:36.217234 10419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:36.217262 10419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:36.218853 10419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:36.218896 10419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:36.218931 10419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:36.220754 10419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:36.220983 10419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:36.221297 10419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:36.226308 10424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.226279 10427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:36.226447 10419 server_base.cc:1034] running on GCE node
W20260927 18:34:36.226308 10425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:36.226791 10419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:36.227075 10419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:36.228240 10419 hybrid_clock.cc:648] HybridClock initialized: now 1790534076228212 us; error 34 us; skew 500 ppm
I20260927 18:34:36.229563 10419 webserver.cc:460] Webserver started at http://127.5.151.193:34965/ using document root <none> and password file <none>
I20260927 18:34:36.229791 10419 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:36.229841 10419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:36.229951 10419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:36.231212 10419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "408758188bcc4a3b80cf744c96da1093"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "54083df2971cdd4ccf557e97e8a16e89"
server_key_iv: "d8bc443688a95a671bd5256c6ad720fe"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.231590 10419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "408758188bcc4a3b80cf744c96da1093"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "54083df2971cdd4ccf557e97e8a16e89"
server_key_iv: "d8bc443688a95a671bd5256c6ad720fe"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.232944 10419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:36.233798 10433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.233976 10419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:36.234059 10419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "408758188bcc4a3b80cf744c96da1093"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "54083df2971cdd4ccf557e97e8a16e89"
server_key_iv: "d8bc443688a95a671bd5256c6ad720fe"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.234141 10419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:36.238238 10419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:36.238476 10419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:36.238646 10419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:36.238840 10419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:36.239141 10419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:36.239199 10419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.239230 10419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:36.239256 10419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.244638 10419 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:41297
I20260927 18:34:36.244688 10546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:41297 every 8 connection(s)
I20260927 18:34:36.244961 10419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:36.247859 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.245572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:57675 (local address 127.5.151.254:38029)
0927 18:34:36.245625 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.245628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.245904 (+ 276us) server_negotiation.cc:407] Connection header received
0927 18:34:36.246376 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.246377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.246384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.246449 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.247611 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.247612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.247613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.247617 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.247632 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.247636 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.247715 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.247785 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.247799 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:34:36.248065 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.245572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38029 (local address 127.5.151.193:57675)
0927 18:34:36.245765 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.245802 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.246269 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.246513 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.246517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.246526 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.247460 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.247461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.247484 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.247509 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.247713 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.247718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.247721 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.247794 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.247846 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260927 18:34:36.248544 10547 heartbeater.cc:347] Connected to a master server at 127.5.151.254:38029
I20260927 18:34:36.248611 10547 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:36.248761 10547 heartbeater.cc:511] Master 127.5.151.254:38029 requested a full tablet report, sending...
I20260927 18:34:36.249181 10359 ts_manager.cc:194] Registered new tserver with Master: 408758188bcc4a3b80cf744c96da1093 (127.5.151.193:41297)
I20260927 18:34:36.249409 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10419
I20260927 18:34:36.249500 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:36.249673 5727 external_mini_cluster.cc:1428] Setting key 7e2217d8bd36f766e57f54bdc28b44a3
I20260927 18:34:36.250437 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38029
--builtin_ntp_servers=127.5.151.212:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:34:36.327108 10550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:36.327325 10550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:36.327359 10550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:36.328431 10550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:36.328492 10550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:36.328531 10550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:36.329780 10550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:36.330020 10550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:36.330286 10550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:36.334822 10556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.334856 10555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.334832 10558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:36.334982 10550 server_base.cc:1034] running on GCE node
I20260927 18:34:36.335242 10550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:36.335440 10550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:36.336571 10550 hybrid_clock.cc:648] HybridClock initialized: now 1790534076336552 us; error 34 us; skew 500 ppm
I20260927 18:34:36.337711 10550 webserver.cc:460] Webserver started at http://127.5.151.194:38877/ using document root <none> and password file <none>
I20260927 18:34:36.337945 10550 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:36.338012 10550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:36.338162 10550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:36.339258 10550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "a39bc3f39f634d68b20dfbfcef520294"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "8238c5b6e467eb36276fdb6de65a5a20"
server_key_iv: "01d6989023950d66590f7137cce8e219"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.339624 10550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "a39bc3f39f634d68b20dfbfcef520294"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "8238c5b6e467eb36276fdb6de65a5a20"
server_key_iv: "01d6989023950d66590f7137cce8e219"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.340886 10550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:34:36.341566 10564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.341725 10550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:36.341812 10550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "a39bc3f39f634d68b20dfbfcef520294"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "8238c5b6e467eb36276fdb6de65a5a20"
server_key_iv: "01d6989023950d66590f7137cce8e219"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.341893 10550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:36.345003 10550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:36.345248 10550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:36.345379 10550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:36.345587 10550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:36.345862 10550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:36.345914 10550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.345976 10550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:36.346004 10550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.351542 10550 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:44581
I20260927 18:34:36.351588 10677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:44581 every 8 connection(s)
I20260927 18:34:36.351868 10550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:36.354310 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.352487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:53669 (local address 127.5.151.254:38029)
0927 18:34:36.352550 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.352552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.352731 (+ 179us) server_negotiation.cc:407] Connection header received
0927 18:34:36.353082 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.353082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.353088 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.353137 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.354060 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.354067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.354067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.354071 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.354085 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.354089 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.354151 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.354192 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.354222 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:34:36.354380 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10550
I20260927 18:34:36.354400 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.352468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38029 (local address 127.5.151.194:53669)
0927 18:34:36.352629 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.352657 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.353010 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.353178 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.353181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.353187 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.353937 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.353937 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.353960 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.353984 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.354147 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.354149 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.354151 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.354199 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.354239 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":92,"threads_started":1}
I20260927 18:34:36.354468 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:36.354683 5727 external_mini_cluster.cc:1428] Setting key a812ef9cce4dc11c0d45f147cc70700a
I20260927 18:34:36.354849 10678 heartbeater.cc:347] Connected to a master server at 127.5.151.254:38029
I20260927 18:34:36.354949 10678 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:36.355085 10678 heartbeater.cc:511] Master 127.5.151.254:38029 requested a full tablet report, sending...
I20260927 18:34:36.355448 10359 ts_manager.cc:194] Registered new tserver with Master: a39bc3f39f634d68b20dfbfcef520294 (127.5.151.194:44581)
I20260927 18:34:36.355634 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38029
--builtin_ntp_servers=127.5.151.212:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:34:36.429411 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:36.429585 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:36.429605 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:36.430784 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:36.430819 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:36.430855 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:36.432113 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:36.432322 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:36.432574 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:36.437021 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.436997 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.437011 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:36.437285 10681 server_base.cc:1034] running on GCE node
I20260927 18:34:36.437482 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:36.437721 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:36.438887 10681 hybrid_clock.cc:648] HybridClock initialized: now 1790534076438877 us; error 33 us; skew 500 ppm
I20260927 18:34:36.440016 10681 webserver.cc:460] Webserver started at http://127.5.151.195:44455/ using document root <none> and password file <none>
I20260927 18:34:36.440243 10681 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:36.440306 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:36.440439 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:36.441406 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "0aec4d8169cda46f50f358db475ef73b"
server_key_iv: "ed7b0e9d53d069176ca0c94df82901c1"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.441752 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "0aec4d8169cda46f50f358db475ef73b"
server_key_iv: "ed7b0e9d53d069176ca0c94df82901c1"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.442997 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:36.443596 10695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.443781 10681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:36.443872 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "0aec4d8169cda46f50f358db475ef73b"
server_key_iv: "ed7b0e9d53d069176ca0c94df82901c1"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.443957 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:36.447363 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:36.447556 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:36.447656 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:36.447789 10681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:36.448074 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:36.448107 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.448125 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:36.448169 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.453699 10681 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:39647
I20260927 18:34:36.453747 10808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:39647 every 8 connection(s)
I20260927 18:34:36.454041 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:36.456526 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.454657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:58937 (local address 127.5.151.254:38029)
0927 18:34:36.454697 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.454699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.454948 (+ 249us) server_negotiation.cc:407] Connection header received
0927 18:34:36.455322 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.455323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.455330 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.455377 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.456340 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.456341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.456341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.456345 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.456358 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.456362 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.456407 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.456463 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.456475 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:34:36.456708 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.454650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38029 (local address 127.5.151.195:58937)
0927 18:34:36.454793 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.454847 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.455246 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.455421 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.455424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.455431 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.456215 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.456215 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.456238 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.456264 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.456404 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.456406 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.456408 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.456462 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.456504 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260927 18:34:36.457171 10809 heartbeater.cc:347] Connected to a master server at 127.5.151.254:38029
I20260927 18:34:36.457258 10809 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:36.457391 10809 heartbeater.cc:511] Master 127.5.151.254:38029 requested a full tablet report, sending...
I20260927 18:34:36.457736 10359 ts_manager.cc:194] Registered new tserver with Master: 1f5c8b4ec7ae4d0ca549736bec74bdca (127.5.151.195:39647)
I20260927 18:34:36.459801 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10681
I20260927 18:34:36.459901 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:36.460057 5727 external_mini_cluster.cc:1428] Setting key 20c667ab43e78e457ad972f16d74dd11
I20260927 18:34:36.460919 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:34:36.463191 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.462164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.5.151.254:38029)
0927 18:34:36.462226 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.462228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.462342 (+ 114us) server_negotiation.cc:407] Connection header received
0927 18:34:36.462416 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.462417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.462421 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.462469 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.462990 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.462992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.462992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.462996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.463008 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.463012 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.463064 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.463113 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.463128 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:34:36.464731 10359 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:36.466189 10359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:36.470005 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.468277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.5.151.194:44581)
0927 18:34:36.468429 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.468433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.468672 (+ 239us) server_negotiation.cc:407] Connection header received
0927 18:34:36.468725 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.468728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.468739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.468814 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.469474 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.469480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.469480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.469484 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.469516 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.469539 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.469603 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.469903 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.469921 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260927 18:34:36.469985 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.468131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:41297 (local address 127.0.0.1:53380)
0927 18:34:36.468366 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.468404 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.468466 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.468665 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.468671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.468676 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.469522 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.469522 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.469535 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.469544 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.469801 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.469807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.469810 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.469878 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.469893 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":54,"threads_started":1}
I20260927 18:34:36.470057 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.468209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.5.151.193:41297)
0927 18:34:36.468471 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.468475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.468508 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:34:36.468558 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.468560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.468570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.468647 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.469634 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.469642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.469642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.469647 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.469684 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.469751 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.469807 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.469903 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.469922 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":68,"threads_started":1}
I20260927 18:34:36.469983 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.468190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:44581 (local address 127.0.0.1:40104)
0927 18:34:36.468596 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.468604 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.468669 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.468823 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.468824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.468828 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.469366 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.469367 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.469380 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.469401 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.469832 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.469833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.469834 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.469885 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.469895 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":81,"threads_started":1}
I20260927 18:34:36.470307 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.468311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:39647 (local address 127.0.0.1:59022)
0927 18:34:36.468978 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.468989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.469037 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.469202 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.469203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.469205 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.469523 (+ 318us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.469523 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.469530 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.469541 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.470178 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.470179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.470180 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.470238 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.470250 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":67,"threads_started":1}
I20260927 18:34:36.470502 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.468451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.5.151.195:39647)
0927 18:34:36.468591 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.468596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.469092 (+ 496us) server_negotiation.cc:407] Connection header received
0927 18:34:36.469115 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.469118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.469125 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.469198 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.470057 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.470062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.470063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.470067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.470101 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.470120 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.470182 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.470393 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.470412 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260927 18:34:36.470784 10481 tablet_service.cc:1467] Processing CreateTablet for tablet 671ace88a7304b4d931e4f2b5eefe089 (DEFAULT_TABLE table=test-table [id=86f952b13a544a8fbda02e248ebf885f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:36.471192 10481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 671ace88a7304b4d931e4f2b5eefe089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:36.471108 10612 tablet_service.cc:1467] Processing CreateTablet for tablet 671ace88a7304b4d931e4f2b5eefe089 (DEFAULT_TABLE table=test-table [id=86f952b13a544a8fbda02e248ebf885f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:36.471415 10612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 671ace88a7304b4d931e4f2b5eefe089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:36.473681 10823 tablet_bootstrap.cc:492] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093: Bootstrap starting.
I20260927 18:34:36.474020 10743 tablet_service.cc:1467] Processing CreateTablet for tablet 671ace88a7304b4d931e4f2b5eefe089 (DEFAULT_TABLE table=test-table [id=86f952b13a544a8fbda02e248ebf885f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:36.474304 10743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 671ace88a7304b4d931e4f2b5eefe089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:36.474314 10823 tablet_bootstrap.cc:654] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:36.474628 10823 log.cc:824] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:36.474712 10824 tablet_bootstrap.cc:492] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Bootstrap starting.
I20260927 18:34:36.475279 10824 tablet_bootstrap.cc:654] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:36.475343 10823 tablet_bootstrap.cc:492] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093: No bootstrap required, opened a new log
I20260927 18:34:36.475410 10823 ts_tablet_manager.cc:1397] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:34:36.475597 10824 log.cc:824] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:36.476305 10824 tablet_bootstrap.cc:492] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: No bootstrap required, opened a new log
I20260927 18:34:36.476370 10824 ts_tablet_manager.cc:1397] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:36.477025 10827 tablet_bootstrap.cc:492] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Bootstrap starting.
I20260927 18:34:36.477545 10827 tablet_bootstrap.cc:654] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:36.477836 10827 log.cc:824] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:36.477882 10823 raft_consensus.cc:348] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.478048 10823 raft_consensus.cc:374] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:36.478096 10823 raft_consensus.cc:729] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408758188bcc4a3b80cf744c96da1093, State: Initialized, Role: FOLLOWER
I20260927 18:34:36.478220 10823 consensus_queue.cc:260] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.478490 10823 ts_tablet_manager.cc:1428] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:34:36.478627 10547 heartbeater.cc:503] Master 127.5.151.254:38029 was elected leader, sending a full tablet report...
I20260927 18:34:36.478660 10827 tablet_bootstrap.cc:492] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: No bootstrap required, opened a new log
I20260927 18:34:36.478722 10827 ts_tablet_manager.cc:1397] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:36.478832 10824 raft_consensus.cc:348] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.478964 10824 raft_consensus.cc:374] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:36.478986 10824 raft_consensus.cc:729] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39bc3f39f634d68b20dfbfcef520294, State: Initialized, Role: FOLLOWER
I20260927 18:34:36.479081 10824 consensus_queue.cc:260] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.479310 10824 ts_tablet_manager.cc:1428] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:34:36.479321 10678 heartbeater.cc:503] Master 127.5.151.254:38029 was elected leader, sending a full tablet report...
I20260927 18:34:36.480991 10827 raft_consensus.cc:348] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.481122 10827 raft_consensus.cc:374] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:36.481165 10827 raft_consensus.cc:729] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5c8b4ec7ae4d0ca549736bec74bdca, State: Initialized, Role: FOLLOWER
I20260927 18:34:36.481267 10827 consensus_queue.cc:260] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.481521 10827 ts_tablet_manager.cc:1428] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:34:36.481583 10809 heartbeater.cc:503] Master 127.5.151.254:38029 was elected leader, sending a full tablet report...
W20260927 18:34:36.495426 10548 tablet.cc:2369] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:36.599581 10829 raft_consensus.cc:482] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:36.599704 10829 raft_consensus.cc:504] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.600018 10829 leader_election.cc:288] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5c8b4ec7ae4d0ca549736bec74bdca (127.5.151.195:39647), a39bc3f39f634d68b20dfbfcef520294 (127.5.151.194:44581)
I20260927 18:34:36.601330 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.600327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:38931 (local address 127.5.151.194:44581)
0927 18:34:36.600385 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.600387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.600395 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:36.600418 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.600418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.600422 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.600482 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.601124 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.601125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.601126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.601130 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.601146 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.601151 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.601202 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.601257 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.601269 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:34:36.601330 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.600246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:44581 (local address 127.5.151.193:38931)
0927 18:34:36.600293 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.600304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.600342 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.600505 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.600506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.600509 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.601028 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.601029 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.601036 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.601046 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.601198 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.601198 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.601199 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.601248 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.601263 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 18:34:36.601660 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.600247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:39647 (local address 127.5.151.193:53967)
0927 18:34:36.600567 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.600575 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.600613 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.600732 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.600733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.600735 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:36.601196 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:34:36.601197 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:36.601203 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:34:36.601214 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:36.601513 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:36.601513 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:36.601514 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 18:34:36.601566 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 18:34:36.601580 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":152,"thread_start_us":114,"threads_started":1}
I20260927 18:34:36.601651 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.600322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:53967 (local address 127.5.151.195:39647)
0927 18:34:36.600400 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.600402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.600631 (+ 229us) server_negotiation.cc:407] Connection header received
0927 18:34:36.600659 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.600660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.600666 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.600721 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.601298 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.601300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.601300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.601305 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.601320 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.601324 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.601396 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.601578 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.601590 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:34:36.601666 10632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671ace88a7304b4d931e4f2b5eefe089" candidate_uuid: "408758188bcc4a3b80cf744c96da1093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39bc3f39f634d68b20dfbfcef520294" is_pre_election: true
I20260927 18:34:36.601840 10632 raft_consensus.cc:2454] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 408758188bcc4a3b80cf744c96da1093 in term 0.
I20260927 18:34:36.601987 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671ace88a7304b4d931e4f2b5eefe089" candidate_uuid: "408758188bcc4a3b80cf744c96da1093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" is_pre_election: true
I20260927 18:34:36.602064 10437 leader_election.cc:302] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408758188bcc4a3b80cf744c96da1093, a39bc3f39f634d68b20dfbfcef520294; no voters:
I20260927 18:34:36.602129 10763 raft_consensus.cc:2454] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 408758188bcc4a3b80cf744c96da1093 in term 0.
I20260927 18:34:36.602176 10829 raft_consensus.cc:2781] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:36.602257 10829 raft_consensus.cc:482] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:36.602288 10829 raft_consensus.cc:3037] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:34:36.602347 10679 tablet.cc:2369] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:36.603083 10829 raft_consensus.cc:504] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.603217 10829 leader_election.cc:288] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5c8b4ec7ae4d0ca549736bec74bdca (127.5.151.195:39647), a39bc3f39f634d68b20dfbfcef520294 (127.5.151.194:44581)
I20260927 18:34:36.603363 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671ace88a7304b4d931e4f2b5eefe089" candidate_uuid: "408758188bcc4a3b80cf744c96da1093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca"
I20260927 18:34:36.603381 10632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671ace88a7304b4d931e4f2b5eefe089" candidate_uuid: "408758188bcc4a3b80cf744c96da1093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39bc3f39f634d68b20dfbfcef520294"
I20260927 18:34:36.603461 10632 raft_consensus.cc:3037] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:36.603466 10763 raft_consensus.cc:3037] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:36.604135 10632 raft_consensus.cc:2454] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 408758188bcc4a3b80cf744c96da1093 in term 1.
I20260927 18:34:36.604206 10763 raft_consensus.cc:2454] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 408758188bcc4a3b80cf744c96da1093 in term 1.
I20260927 18:34:36.604324 10437 leader_election.cc:302] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408758188bcc4a3b80cf744c96da1093, a39bc3f39f634d68b20dfbfcef520294; no voters:
I20260927 18:34:36.604408 10829 raft_consensus.cc:2781] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:36.604521 10829 raft_consensus.cc:686] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [term 1 LEADER]: Becoming Leader. State: Replica: 408758188bcc4a3b80cf744c96da1093, State: Running, Role: LEADER
I20260927 18:34:36.604610 10829 consensus_queue.cc:237] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } }
I20260927 18:34:36.605348 10358 catalog_manager.cc:5495] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 reported cstate change: term changed from 0 to 1, leader changed from <none> to 408758188bcc4a3b80cf744c96da1093 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "408758188bcc4a3b80cf744c96da1093" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "408758188bcc4a3b80cf744c96da1093" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 } health_report { overall_health: UNKNOWN } } }
I20260927 18:34:36.613214 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.612398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.5.151.193:41297)
0927 18:34:36.612439 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.612441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.612449 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:36.612521 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.612521 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.612531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.612578 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.613015 (+ 437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.613016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.613016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.613021 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.613033 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.613036 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.613090 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.613142 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.613155 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:36.614668 10632 raft_consensus.cc:1261] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 1 FOLLOWER]: Refusing update from remote peer 408758188bcc4a3b80cf744c96da1093: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:36.614719 10763 raft_consensus.cc:1261] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 1 FOLLOWER]: Refusing update from remote peer 408758188bcc4a3b80cf744c96da1093: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:36.614979 10833 consensus_queue.cc:1035] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39bc3f39f634d68b20dfbfcef520294" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:36.615101 10829 consensus_queue.cc:1035] T 671ace88a7304b4d931e4f2b5eefe089 P 408758188bcc4a3b80cf744c96da1093 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5c8b4ec7ae4d0ca549736bec74bdca" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:36.617511 10837 mvcc.cc:205] Tried to move back new op lower bound from 7334027577811349504 to 7334027577772896256. Current Snapshot: MvccSnapshot[applied={T|T < 7334027577811349504 or (T in {7334027577811349504})}]
I20260927 18:34:36.618054 10839 mvcc.cc:205] Tried to move back new op lower bound from 7334027577811349504 to 7334027577772896256. Current Snapshot: MvccSnapshot[applied={T|T < 7334027577811349504}]
I20260927 18:34:36.619051 10358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:36.619133 10358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:36.619802 10358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61375dd5d1064397929120ed35aa7a28: Sending DeleteTablet for 3 replicas of tablet 671ace88a7304b4d931e4f2b5eefe089
I20260927 18:34:36.620126 10612 tablet_service.cc:1514] Processing DeleteTablet for tablet 671ace88a7304b4d931e4f2b5eefe089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:36 UTC) from {username='slave'} at 127.0.0.1:40104
I20260927 18:34:36.620133 10481 tablet_service.cc:1514] Processing DeleteTablet for tablet 671ace88a7304b4d931e4f2b5eefe089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:36 UTC) from {username='slave'} at 127.0.0.1:53380
I20260927 18:34:36.620393 10743 tablet_service.cc:1514] Processing DeleteTablet for tablet 671ace88a7304b4d931e4f2b5eefe089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:36 UTC) from {username='slave'} at 127.0.0.1:59022
I20260927 18:34:36.620438 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10419
I20260927 18:34:36.621045 10846 tablet_replica.cc:330] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: stopping tablet replica
I20260927 18:34:36.621140 10846 raft_consensus.cc:2229] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:36.621219 10847 tablet_replica.cc:330] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: stopping tablet replica
I20260927 18:34:36.621248 10846 raft_consensus.cc:2258] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:36.621345 10847 raft_consensus.cc:2229] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:36.621575 10847 raft_consensus.cc:2258] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:36.621668 10846 ts_tablet_manager.cc:1905] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:36.621948 10847 ts_tablet_manager.cc:1905] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:36.623152 10847 ts_tablet_manager.cc:1918] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:36.623234 10847 log.cc:1196] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/671ace88a7304b4d931e4f2b5eefe089
I20260927 18:34:36.623479 10847 ts_tablet_manager.cc:1939] T 671ace88a7304b4d931e4f2b5eefe089 P 1f5c8b4ec7ae4d0ca549736bec74bdca: Deleting consensus metadata
I20260927 18:34:36.623764 10846 ts_tablet_manager.cc:1918] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:36.623832 10846 log.cc:1196] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/671ace88a7304b4d931e4f2b5eefe089
I20260927 18:34:36.623835 10344 catalog_manager.cc:4841] TS 1f5c8b4ec7ae4d0ca549736bec74bdca (127.5.151.195:39647): tablet 671ace88a7304b4d931e4f2b5eefe089 (table test-table [id=86f952b13a544a8fbda02e248ebf885f]) successfully deleted
I20260927 18:34:36.624092 10846 ts_tablet_manager.cc:1939] T 671ace88a7304b4d931e4f2b5eefe089 P a39bc3f39f634d68b20dfbfcef520294: Deleting consensus metadata
I20260927 18:34:36.624470 10346 catalog_manager.cc:4841] TS a39bc3f39f634d68b20dfbfcef520294 (127.5.151.194:44581): tablet 671ace88a7304b4d931e4f2b5eefe089 (table test-table [id=86f952b13a544a8fbda02e248ebf885f]) successfully deleted
W20260927 18:34:36.627393 10344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:41297 (error 108)
I20260927 18:34:36.627450 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10550
I20260927 18:34:36.628436 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.628296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:41297 (local address 127.0.0.1:53406)
0927 18:34:36.628354 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.628398 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 18:34:36.628574 10344 catalog_manager.cc:4584] TS 408758188bcc4a3b80cf744c96da1093 (127.5.151.193:41297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 671ace88a7304b4d931e4f2b5eefe089: Network error: Client connection negotiation failed: client connection to 127.5.151.193:41297: connect: Connection refused (error 111)
I20260927 18:34:36.632493 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10681
I20260927 18:34:36.636318 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10328
2026-09-27T18:34:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (706 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 18:34:36.655110 5727 mini_oidc.cc:152] Starting JWKS server
I20260927 18:34:36.656133 5727 webserver.cc:460] Webserver started at https://127.0.0.1:36475/ using document root <none> and password file <none>
I20260927 18:34:36.656207 5727 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:34:36.656327 5727 webserver.cc:460] Webserver started at http://127.0.0.1:35319/ using document root <none> and password file <none>
2026-09-27T18:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:36Z Disabled control of system clock
I20260927 18:34:36.663895 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38635
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:38635
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36475/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/testchainca.cert with env {}
W20260927 18:34:36.740360 10854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:36.740564 10854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:36.740585 10854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:36.741633 10854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:36.741667 10854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:36.741679 10854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:36.741691 10854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:36.741703 10854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:34:36.741715 10854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:34:36.741726 10854 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36475/jwks.json
I20260927 18:34:36.743163 10854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:38635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36475/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:36.743336 10854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:36.743476 10854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:36.747855 10862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.747865 10860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.747903 10859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:36.748123 10854 server_base.cc:1034] running on GCE node
I20260927 18:34:36.748340 10854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:36.748596 10854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:36.749728 10854 hybrid_clock.cc:648] HybridClock initialized: now 1790534076749722 us; error 29 us; skew 500 ppm
I20260927 18:34:36.756345 10854 webserver.cc:460] Webserver started at http://127.5.151.254:36427/ using document root <none> and password file <none>
I20260927 18:34:36.756593 10854 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:36.756639 10854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:36.756794 10854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:36.757825 10854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "e08faff281db47f0bfcd63b9c3351017"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "f1e053b4bb8f67c910178227337911a3"
server_key_iv: "c18a969e8a237dbbd62d9c114cd41ff2"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.758168 10854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "e08faff281db47f0bfcd63b9c3351017"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "f1e053b4bb8f67c910178227337911a3"
server_key_iv: "c18a969e8a237dbbd62d9c114cd41ff2"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.759821 10854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:36.760480 10868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.760650 10854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:36.760743 10854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "e08faff281db47f0bfcd63b9c3351017"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "f1e053b4bb8f67c910178227337911a3"
server_key_iv: "c18a969e8a237dbbd62d9c114cd41ff2"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.760857 10854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:36.774976 10854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:36.775250 10854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:36.775395 10854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:36.778867 10854 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:38635
I20260927 18:34:36.778900 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:38635 every 8 connection(s)
I20260927 18:34:36.779214 10854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:36.780074 10921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:36.782120 10921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017: Bootstrap starting.
I20260927 18:34:36.782637 10921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:36.782855 10921 log.cc:824] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:36.783432 10921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017: No bootstrap required, opened a new log
I20260927 18:34:36.784871 10921 raft_consensus.cc:348] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08faff281db47f0bfcd63b9c3351017" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38635 } }
I20260927 18:34:36.784997 10921 raft_consensus.cc:374] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:36.785019 10921 raft_consensus.cc:729] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08faff281db47f0bfcd63b9c3351017, State: Initialized, Role: FOLLOWER
I20260927 18:34:36.785111 10921 consensus_queue.cc:260] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08faff281db47f0bfcd63b9c3351017" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38635 } }
I20260927 18:34:36.785159 10921 raft_consensus.cc:388] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:36.785178 10921 raft_consensus.cc:482] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:36.785203 10921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:36.785629 10921 raft_consensus.cc:504] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08faff281db47f0bfcd63b9c3351017" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38635 } }
I20260927 18:34:36.785702 10921 leader_election.cc:302] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e08faff281db47f0bfcd63b9c3351017; no voters:
I20260927 18:34:36.785817 10921 leader_election.cc:288] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:36.785936 10924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:36.786031 10921 sys_catalog.cc:564] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:36.786089 10924 raft_consensus.cc:686] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [term 1 LEADER]: Becoming Leader. State: Replica: e08faff281db47f0bfcd63b9c3351017, State: Running, Role: LEADER
I20260927 18:34:36.786216 10924 consensus_queue.cc:237] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08faff281db47f0bfcd63b9c3351017" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38635 } }
I20260927 18:34:36.786623 10925 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e08faff281db47f0bfcd63b9c3351017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08faff281db47f0bfcd63b9c3351017" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38635 } } }
I20260927 18:34:36.786710 10925 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:36.786696 10926 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e08faff281db47f0bfcd63b9c3351017. Latest consensus state: current_term: 1 leader_uuid: "e08faff281db47f0bfcd63b9c3351017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08faff281db47f0bfcd63b9c3351017" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38635 } } }
I20260927 18:34:36.786746 10926 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:36.786899 10933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:36.787393 10933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:36.787595 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10854
I20260927 18:34:36.787696 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:36.787856 5727 external_mini_cluster.cc:1428] Setting key dbca799e91a54de33a3da80d19533b89
I20260927 18:34:36.788993 10933 catalog_manager.cc:1347] Generated new cluster ID: 3669edb629dd4513be6e6242a994cfb3
I20260927 18:34:36.789041 10933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:36.793848 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.788296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.5.151.254:38635)
0927 18:34:36.788494 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.788502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.788534 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:34:36.788703 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.788709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.788771 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.788912 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.789212 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.790594 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.791705 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.792342 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.792474 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:36.793096 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.793099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.793100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.793294 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.793332 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.793345 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.793414 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.793481 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.793643 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260927 18:34:36.794785 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:34:36.795517 10933 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:36.796087 10933 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:36.802263 10933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e08faff281db47f0bfcd63b9c3351017: Generated new TSK 0
I20260927 18:34:36.802424 10933 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:34:36.823100 10951 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 18:34:36.823187 10951 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:34:36.823397 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.808973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.5.151.254:38635)
0927 18:34:36.809018 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.809020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.809152 (+ 132us) server_negotiation.cc:407] Connection header received
0927 18:34:36.809242 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.809242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.809252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.809307 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:34:36.809436 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.809882 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.811036 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.811205 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.811266 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:36.811466 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:34:36.823213 (+ 11747us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:34:36.823350 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 18:34:36.823460 10943 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 18:34:36.823752 10950 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:36.808869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38635 (local address 127.0.0.1:41656)
0927 18:34:36.809069 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:36.809083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:36.809157 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:36.809312 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:36.809313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:36.809315 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:34:36.809377 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:36.809379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.809980 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.809981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:36.810960 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:36.810963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.811286 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.811287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:36.811349 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:36.811368 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:34:36.823349 (+ 11981us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:34:36.823466 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.151.254:38635: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.151.254:38635: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 18:34:36.824191 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10854
2026-09-27T18:34:36Z chronyd exiting
I20260927 18:34:36.859702 5727 test_util.cc:175] -----------------------------------------------
I20260927 18:34:36.859817 5727 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534055081918-5727-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 18:34:36.860523 5727 mini_oidc.cc:152] Starting JWKS server
I20260927 18:34:36.861208 5727 webserver.cc:460] Webserver started at https://127.0.0.1:40319/ using document root <none> and password file <none>
I20260927 18:34:36.861271 5727 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:34:36.861438 5727 webserver.cc:460] Webserver started at http://127.0.0.1:33351/ using document root <none> and password file <none>
2026-09-27T18:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:36Z Disabled control of system clock
I20260927 18:34:36.868224 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34041
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:34041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40319/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/testchainca.cert with env {}
W20260927 18:34:36.943909 10957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:36.944126 10957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:36.944161 10957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:36.945257 10957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:36.945319 10957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:36.945340 10957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:36.945369 10957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:36.945396 10957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:34:36.945426 10957 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:34:36.945451 10957 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40319/jwks.json
I20260927 18:34:36.946929 10957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:34041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40319/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:36.947176 10957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:36.947371 10957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:36.951681 10965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.951675 10963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:36.951675 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:36.951894 10957 server_base.cc:1034] running on GCE node
I20260927 18:34:36.952131 10957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:36.952437 10957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:36.953598 10957 hybrid_clock.cc:648] HybridClock initialized: now 1790534076953587 us; error 41 us; skew 500 ppm
I20260927 18:34:36.954823 10957 webserver.cc:460] Webserver started at http://127.5.151.254:36897/ using document root <none> and password file <none>
I20260927 18:34:36.955062 10957 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:36.955143 10957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:36.955267 10957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:36.956182 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "63950bf746424a418d2e047dbd4e53d3"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "6552726153fe0264b89099ef8a0b3251"
server_key_iv: "b95d360e1866f6686bfb3a99dd28ba58"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.956535 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "63950bf746424a418d2e047dbd4e53d3"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "6552726153fe0264b89099ef8a0b3251"
server_key_iv: "b95d360e1866f6686bfb3a99dd28ba58"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.957774 10957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:36.958474 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:36.958657 10957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:36.958753 10957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "63950bf746424a418d2e047dbd4e53d3"
format_stamp: "Formatted at 2026-09-27 18:34:36 on dist-test-slave-19nl"
server_key: "6552726153fe0264b89099ef8a0b3251"
server_key_iv: "b95d360e1866f6686bfb3a99dd28ba58"
server_key_version: "encryptionkey@0"
I20260927 18:34:36.958830 10957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:36.967962 10957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:36.968220 10957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:36.968370 10957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:36.972275 10957 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:34041
I20260927 18:34:36.972302 11023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:34041 every 8 connection(s)
I20260927 18:34:36.972602 10957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:36.973510 11024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:36.975548 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3: Bootstrap starting.
I20260927 18:34:36.976011 11024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:36.976230 11024 log.cc:824] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:36.976801 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3: No bootstrap required, opened a new log
I20260927 18:34:36.978235 11024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63950bf746424a418d2e047dbd4e53d3" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34041 } }
I20260927 18:34:36.978351 11024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:36.978372 11024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63950bf746424a418d2e047dbd4e53d3, State: Initialized, Role: FOLLOWER
I20260927 18:34:36.978458 11024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63950bf746424a418d2e047dbd4e53d3" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34041 } }
I20260927 18:34:36.978513 11024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:36.978531 11024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:36.978611 11024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:36.979053 11024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63950bf746424a418d2e047dbd4e53d3" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34041 } }
I20260927 18:34:36.979130 11024 leader_election.cc:302] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63950bf746424a418d2e047dbd4e53d3; no voters:
I20260927 18:34:36.979230 11024 leader_election.cc:288] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:36.979317 11027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:36.979490 11027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [term 1 LEADER]: Becoming Leader. State: Replica: 63950bf746424a418d2e047dbd4e53d3, State: Running, Role: LEADER
I20260927 18:34:36.979516 11024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:36.979581 11027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63950bf746424a418d2e047dbd4e53d3" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34041 } }
I20260927 18:34:36.980069 11029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63950bf746424a418d2e047dbd4e53d3. Latest consensus state: current_term: 1 leader_uuid: "63950bf746424a418d2e047dbd4e53d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63950bf746424a418d2e047dbd4e53d3" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34041 } } }
I20260927 18:34:36.980278 11029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:36.979951 11028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63950bf746424a418d2e047dbd4e53d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63950bf746424a418d2e047dbd4e53d3" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34041 } } }
I20260927 18:34:36.980542 11028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:36.980592 11037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:36.981010 11037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:36.981875 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 10957
I20260927 18:34:36.981966 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:36.982164 5727 external_mini_cluster.cc:1428] Setting key 4f78584b79d4284e92bab3c5a021187b
I20260927 18:34:36.982738 11037 catalog_manager.cc:1347] Generated new cluster ID: 86c6ba2999db47dd9b4a2b7482bbf9ff
I20260927 18:34:36.982823 11037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:36.987951 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:36.982766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.5.151.254:34041)
0927 18:34:36.982934 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:36.982943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:36.982972 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:34:36.983082 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:36.983089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:36.983150 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:36.983286 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:36.983505 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.984853 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.985881 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:36.986509 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:36.986667 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:36.987291 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:36.987294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:36.987295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:36.987492 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:36.987535 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:36.987549 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:36.987619 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:36.987689 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 18:34:36.987761 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260927 18:34:36.988996 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:34:36.993151 11037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:36.993602 11037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:37.000010 11037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63950bf746424a418d2e047dbd4e53d3: Generated new TSK 0
I20260927 18:34:37.000170 11037 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:34:37.022198 11054 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 18:34:37.022269 11054 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:34:37.022532 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:37.008793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.5.151.254:34041)
0927 18:34:37.008846 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:37.008848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:37.008940 (+ 92us) server_negotiation.cc:407] Connection header received
0927 18:34:37.008976 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:37.008977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:37.008993 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:37.009068 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:34:37.009152 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.009933 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.010687 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.011016 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.011083 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.011209 (+ 126us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:34:37.022354 (+ 11145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:34:37.022471 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33422: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 18:34:37.022605 11047 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33422: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 18:34:37.022655 11053 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:37.008696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34041 (local address 127.0.0.1:33422)
0927 18:34:37.008869 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:37.008879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:37.008935 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:37.009070 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:37.009071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:37.009073 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:34:37.009111 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:37.009112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.010009 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.010010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:37.010611 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:37.010613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.011089 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.011089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:37.011126 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.011130 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:34:37.022453 (+ 11323us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:34:37.022523 (+ 70us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.151.254:34041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260927 18:34:37.022899 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 10957
2026-09-27T18:34:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (199 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 18:34:37.060050 5727 mini_oidc.cc:152] Starting JWKS server
I20260927 18:34:37.060668 5727 webserver.cc:460] Webserver started at https://127.0.0.1:42425/ using document root <none> and password file <none>
I20260927 18:34:37.060730 5727 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:34:37.060895 5727 webserver.cc:460] Webserver started at http://127.0.0.1:42019/ using document root <none> and password file <none>
2026-09-27T18:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:37Z Disabled control of system clock
I20260927 18:34:37.067785 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:37361
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:37361
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:34:37.145918 11060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:37.146104 11060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:37.146124 11060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:37.147761 11060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:37.147801 11060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:37.147814 11060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:37.147826 11060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:37.147871 11060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:34:37.147897 11060 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:34:37.147923 11060 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42425/jwks.json
I20260927 18:34:37.149250 11060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:37361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:37.149485 11060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:37.149756 11060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:37.154176 11065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:37.154196 11068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:37.154182 11066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:37.154237 11060 server_base.cc:1034] running on GCE node
I20260927 18:34:37.154637 11060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:37.154914 11060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:37.156066 11060 hybrid_clock.cc:648] HybridClock initialized: now 1790534077156043 us; error 36 us; skew 500 ppm
I20260927 18:34:37.157169 11060 webserver.cc:460] Webserver started at http://127.5.151.254:38961/ using document root <none> and password file <none>
I20260927 18:34:37.157394 11060 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:37.157457 11060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:37.157613 11060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:37.158620 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "305fdf09dd6d459a80653918c9b216bd"
format_stamp: "Formatted at 2026-09-27 18:34:37 on dist-test-slave-19nl"
server_key: "dd88eec29e6995586d6030757854dab4"
server_key_iv: "77ed9f0108eef7ebdbfe60c23f949306"
server_key_version: "encryptionkey@0"
I20260927 18:34:37.158977 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "305fdf09dd6d459a80653918c9b216bd"
format_stamp: "Formatted at 2026-09-27 18:34:37 on dist-test-slave-19nl"
server_key: "dd88eec29e6995586d6030757854dab4"
server_key_iv: "77ed9f0108eef7ebdbfe60c23f949306"
server_key_version: "encryptionkey@0"
I20260927 18:34:37.160166 11060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:37.160851 11074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:37.161049 11060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:37.161154 11060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "305fdf09dd6d459a80653918c9b216bd"
format_stamp: "Formatted at 2026-09-27 18:34:37 on dist-test-slave-19nl"
server_key: "dd88eec29e6995586d6030757854dab4"
server_key_iv: "77ed9f0108eef7ebdbfe60c23f949306"
server_key_version: "encryptionkey@0"
I20260927 18:34:37.161226 11060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:37.172755 11060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:37.172999 11060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:37.173134 11060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:37.176648 11060 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:37361
I20260927 18:34:37.176712 11126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:37361 every 8 connection(s)
I20260927 18:34:37.177021 11060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:37.177608 11127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:37.180054 11127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd: Bootstrap starting.
I20260927 18:34:37.180557 11127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:37.180819 11127 log.cc:824] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:37.181337 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11060
I20260927 18:34:37.181430 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:37.181545 11127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd: No bootstrap required, opened a new log
I20260927 18:34:37.181636 5727 external_mini_cluster.cc:1428] Setting key f7a2c4e8b443bf72474a1a5f527ef09e
I20260927 18:34:37.183799 11127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305fdf09dd6d459a80653918c9b216bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37361 } }
I20260927 18:34:37.183912 11127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:37.183933 11127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305fdf09dd6d459a80653918c9b216bd, State: Initialized, Role: FOLLOWER
I20260927 18:34:37.184062 11127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305fdf09dd6d459a80653918c9b216bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37361 } }
I20260927 18:34:37.184119 11127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:37.184166 11127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:37.184216 11127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:37.184707 11127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305fdf09dd6d459a80653918c9b216bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37361 } }
I20260927 18:34:37.184813 11127 leader_election.cc:302] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 305fdf09dd6d459a80653918c9b216bd; no voters:
I20260927 18:34:37.185006 11132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:37.184993 11127 leader_election.cc:288] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:37.185153 11132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [term 1 LEADER]: Becoming Leader. State: Replica: 305fdf09dd6d459a80653918c9b216bd, State: Running, Role: LEADER
I20260927 18:34:37.185228 11132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305fdf09dd6d459a80653918c9b216bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37361 } }
I20260927 18:34:37.185303 11127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:37.185552 11132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "305fdf09dd6d459a80653918c9b216bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305fdf09dd6d459a80653918c9b216bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37361 } } }
I20260927 18:34:37.185631 11132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:37.185562 11134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 305fdf09dd6d459a80653918c9b216bd. Latest consensus state: current_term: 1 leader_uuid: "305fdf09dd6d459a80653918c9b216bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305fdf09dd6d459a80653918c9b216bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37361 } } }
I20260927 18:34:37.185788 11134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:37.185846 11137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:37.186393 11137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:37.188129 11137 catalog_manager.cc:1347] Generated new cluster ID: 29e290f0a3a645ecb71dc553bf794bcc
I20260927 18:34:37.188221 11137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:37.188293 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:37.182124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.5.151.254:37361)
0927 18:34:37.182309 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:37.182317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:37.182345 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:34:37.182832 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:37.182838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:37.182890 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:37.183030 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:37.183226 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.184115 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.184917 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.185473 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.185869 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.187579 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:37.187582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:37.187583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:37.187764 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:37.187800 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:37.187814 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:37.187880 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:37.187975 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 18:34:37.188053 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260927 18:34:37.191390 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:34:37.195124 11137 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:37.195546 11137 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:37.199177 11137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 305fdf09dd6d459a80653918c9b216bd: Generated new TSK 0
I20260927 18:34:37.199331 11137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:34:37.208645 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:37.205169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.5.151.254:37361)
0927 18:34:37.205225 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:37.205227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:37.205344 (+ 117us) server_negotiation.cc:407] Connection header received
0927 18:34:37.205375 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:37.205376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:37.205387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:37.205452 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:34:37.205575 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.206289 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.207045 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.207381 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.207450 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.207587 (+ 137us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:34:37.208446 (+ 859us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:34:37.208587 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 18:34:37.208693 11131 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 18:34:37.208745 11156 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:37.205112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:37361 (local address 127.0.0.1:35820)
0927 18:34:37.205269 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:37.205278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:37.205334 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:37.205465 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:37.205466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:37.205467 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:34:37.205520 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:37.205521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.206368 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.206369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:37.206975 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:37.206977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.207454 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.207454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:37.207501 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.207505 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:34:37.208561 (+ 1056us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:34:37.208639 (+ 78us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.151.254:37361: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42425/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260927 18:34:37.208958 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11060
E20260927 18:34:37.220827 11157 webserver.cc:533] Webserver: sslize failed:
E20260927 18:34:37.220882 11157 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T18:34:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (187 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T18:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:37Z Disabled control of system clock
I20260927 18:34:37.254356 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38033
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:38033
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/testchainca.cert with env {}
W20260927 18:34:37.328850 11161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:37.329066 11161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:37.329102 11161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:37.330263 11161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:37.330317 11161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:37.330336 11161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:37.330356 11161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:37.330385 11161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:34:37.330413 11161 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:34:37.330436 11161 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 18:34:37.331841 11161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:38033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:37.332091 11161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:37.332289 11161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:37.336683 11167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:37.336676 11169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:37.336725 11161 server_base.cc:1034] running on GCE node
W20260927 18:34:37.336696 11166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:37.337100 11161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:37.337394 11161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:37.338584 11161 hybrid_clock.cc:648] HybridClock initialized: now 1790534077338532 us; error 41 us; skew 500 ppm
I20260927 18:34:37.339653 11161 webserver.cc:460] Webserver started at http://127.5.151.254:35467/ using document root <none> and password file <none>
I20260927 18:34:37.339876 11161 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:37.339939 11161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:37.340077 11161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:37.341101 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "a746bc4af82e497284931cd48851e6fe"
format_stamp: "Formatted at 2026-09-27 18:34:37 on dist-test-slave-19nl"
server_key: "8d9787712b6f927a12c09efd985e5cf0"
server_key_iv: "7546518b7fe6278b8540cee5d38a1862"
server_key_version: "encryptionkey@0"
I20260927 18:34:37.341436 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "a746bc4af82e497284931cd48851e6fe"
format_stamp: "Formatted at 2026-09-27 18:34:37 on dist-test-slave-19nl"
server_key: "8d9787712b6f927a12c09efd985e5cf0"
server_key_iv: "7546518b7fe6278b8540cee5d38a1862"
server_key_version: "encryptionkey@0"
I20260927 18:34:37.342653 11161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:37.343333 11175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:37.343497 11161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:37.343590 11161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "a746bc4af82e497284931cd48851e6fe"
format_stamp: "Formatted at 2026-09-27 18:34:37 on dist-test-slave-19nl"
server_key: "8d9787712b6f927a12c09efd985e5cf0"
server_key_iv: "7546518b7fe6278b8540cee5d38a1862"
server_key_version: "encryptionkey@0"
I20260927 18:34:37.343672 11161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:37.356575 11161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:37.356804 11161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:37.356930 11161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:37.360394 11161 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:38033
I20260927 18:34:37.360420 11227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:38033 every 8 connection(s)
I20260927 18:34:37.360718 11161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:37.361512 11228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:37.363444 11228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe: Bootstrap starting.
I20260927 18:34:37.363999 11228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:37.364265 11228 log.cc:824] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:37.364842 11228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe: No bootstrap required, opened a new log
I20260927 18:34:37.366349 11228 raft_consensus.cc:348] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a746bc4af82e497284931cd48851e6fe" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38033 } }
I20260927 18:34:37.366464 11228 raft_consensus.cc:374] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:37.366510 11228 raft_consensus.cc:729] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a746bc4af82e497284931cd48851e6fe, State: Initialized, Role: FOLLOWER
I20260927 18:34:37.366686 11228 consensus_queue.cc:260] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a746bc4af82e497284931cd48851e6fe" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38033 } }
I20260927 18:34:37.366765 11228 raft_consensus.cc:388] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:37.366802 11228 raft_consensus.cc:482] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:37.366848 11228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:37.367334 11228 raft_consensus.cc:504] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a746bc4af82e497284931cd48851e6fe" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38033 } }
I20260927 18:34:37.367435 11228 leader_election.cc:302] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a746bc4af82e497284931cd48851e6fe; no voters:
I20260927 18:34:37.367595 11228 leader_election.cc:288] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:37.367638 11231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:37.367787 11231 raft_consensus.cc:686] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [term 1 LEADER]: Becoming Leader. State: Replica: a746bc4af82e497284931cd48851e6fe, State: Running, Role: LEADER
I20260927 18:34:37.367859 11228 sys_catalog.cc:564] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:37.367858 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11161
I20260927 18:34:37.367877 11231 consensus_queue.cc:237] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a746bc4af82e497284931cd48851e6fe" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38033 } }
I20260927 18:34:37.367945 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:37.368137 5727 external_mini_cluster.cc:1428] Setting key a7bdad5b0145b85038eab4d7b27476da
I20260927 18:34:37.368160 11232 sys_catalog.cc:455] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a746bc4af82e497284931cd48851e6fe. Latest consensus state: current_term: 1 leader_uuid: "a746bc4af82e497284931cd48851e6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a746bc4af82e497284931cd48851e6fe" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38033 } } }
I20260927 18:34:37.368238 11233 sys_catalog.cc:455] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a746bc4af82e497284931cd48851e6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a746bc4af82e497284931cd48851e6fe" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38033 } } }
I20260927 18:34:37.368296 11233 sys_catalog.cc:458] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:37.368235 11232 sys_catalog.cc:458] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:37.368526 11237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:37.369203 11237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:37.370743 11237 catalog_manager.cc:1347] Generated new cluster ID: fdaab89d7a3e42f4ada50e0e8c5aa67c
I20260927 18:34:37.370795 11237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:37.373633 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:37.368904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.5.151.254:38033)
0927 18:34:37.369120 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:37.369130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:37.369161 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:34:37.369331 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:37.369338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:37.369411 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:37.369609 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:37.369797 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.371061 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.371811 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.372228 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.372362 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.372982 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:37.372985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:37.372985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:34:37.373148 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:37.373177 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:34:37.373186 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:37.373260 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:37.373332 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:34:37.373493 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260927 18:34:37.374469 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:34:37.387769 11237 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:37.388230 11237 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:37.399766 11237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a746bc4af82e497284931cd48851e6fe: Generated new TSK 0
I20260927 18:34:37.399919 11237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:34:37.940604 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:37.408847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.5.151.254:38033)
0927 18:34:37.408905 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:37.408907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:37.409011 (+ 104us) server_negotiation.cc:407] Connection header received
0927 18:34:37.409083 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:37.409083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:37.409095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:37.409208 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:34:37.409343 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.409806 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.410563 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.410750 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.410820 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.411031 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:34:37.940308 (+529277us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:34:37.940528 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42080: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 18:34:37.940686 11247 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42080: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 18:34:37.940686 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:37.408769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38033 (local address 127.0.0.1:42080)
0927 18:34:37.408947 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:37.408961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:37.409027 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:37.409208 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:37.409209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:37.409211 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:34:37.409271 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:37.409273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.409886 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.409887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:37.410453 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:37.410454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:37.410826 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:37.410827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:37.410883 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:37.410890 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:34:37.940451 (+529561us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:34:37.940573 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.151.254:38033: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260927 18:34:37.940958 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11161
2026-09-27T18:34:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (710 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:38 dist-test-slave-19nl krb5kdc[11269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:38 dist-test-slave-19nl krb5kdc[11269](info): set up 2 sockets
Sep 27 18:34:38 dist-test-slave-19nl krb5kdc[11269](info): commencing operation
krb5kdc: starting...
W20260927 18:34:39.122516 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:39.226145 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39139
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:39139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:39.300405 11282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:39.300575 11282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:39.300595 11282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:39.301741 11282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:39.301780 11282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:39.301795 11282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:39.301841 11282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:39.301864 11282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:39.303335 11282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:39139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:39.303563 11282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:39.303771 11282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:39.308235 11288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:39.308239 11287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:39.308331 11282 server_base.cc:1034] running on GCE node
W20260927 18:34:39.308234 11290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:39.308642 11282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:39.308933 11282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:39.310092 11282 hybrid_clock.cc:648] HybridClock initialized: now 1790534079310076 us; error 45 us; skew 500 ppm
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.312400 11282 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:39.312747 11282 webserver.cc:460] Webserver started at http://127.5.151.254:45557/ using document root <none> and password file <none>
I20260927 18:34:39.312920 11282 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:39.312956 11282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:39.313035 11282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:39.313957 11282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "5d2d13f487c041eba41aaec170ac83df"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "006ee6adf41cf7f59362eca672546c2f"
server_key_iv: "8425a84cb0a914a65ecbb702abc5aba5"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.314260 11282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "5d2d13f487c041eba41aaec170ac83df"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "006ee6adf41cf7f59362eca672546c2f"
server_key_iv: "8425a84cb0a914a65ecbb702abc5aba5"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.315522 11282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:39.316169 11297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.316314 11282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:39.316394 11282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "5d2d13f487c041eba41aaec170ac83df"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "006ee6adf41cf7f59362eca672546c2f"
server_key_iv: "8425a84cb0a914a65ecbb702abc5aba5"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.316443 11282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:39.324492 11282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:39.326392 11282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:39.326501 11282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:39.330071 11282 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:39139
I20260927 18:34:39.330161 11349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:39139 every 8 connection(s)
I20260927 18:34:39.330431 11282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:39.331283 11350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:39.333323 11350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df: Bootstrap starting.
I20260927 18:34:39.333873 11350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:39.334148 11350 log.cc:824] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:39.334807 11350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df: No bootstrap required, opened a new log
I20260927 18:34:39.336277 11350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2d13f487c041eba41aaec170ac83df" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39139 } }
I20260927 18:34:39.336433 11350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:39.336477 11350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2d13f487c041eba41aaec170ac83df, State: Initialized, Role: FOLLOWER
I20260927 18:34:39.336668 11350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2d13f487c041eba41aaec170ac83df" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39139 } }
I20260927 18:34:39.336771 11350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:39.336812 11350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:39.336864 11350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:39.337343 11350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2d13f487c041eba41aaec170ac83df" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39139 } }
I20260927 18:34:39.337456 11350 leader_election.cc:302] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d2d13f487c041eba41aaec170ac83df; no voters:
I20260927 18:34:39.337627 11350 leader_election.cc:288] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:39.337678 11353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:39.337829 11353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [term 1 LEADER]: Becoming Leader. State: Replica: 5d2d13f487c041eba41aaec170ac83df, State: Running, Role: LEADER
I20260927 18:34:39.337879 11350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:39.337970 11353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2d13f487c041eba41aaec170ac83df" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39139 } }
I20260927 18:34:39.338323 11355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2d13f487c041eba41aaec170ac83df. Latest consensus state: current_term: 1 leader_uuid: "5d2d13f487c041eba41aaec170ac83df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2d13f487c041eba41aaec170ac83df" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39139 } } }
I20260927 18:34:39.338425 11354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d2d13f487c041eba41aaec170ac83df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2d13f487c041eba41aaec170ac83df" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39139 } } }
I20260927 18:34:39.338455 11355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:39.338523 11354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:39.338909 11362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:39.339377 11362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:39.339730 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11282
I20260927 18:34:39.339856 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:39.340085 5727 external_mini_cluster.cc:1428] Setting key 2a44cc87de36dddfb948c68c587e4605
I20260927 18:34:39.341066 11362 catalog_manager.cc:1347] Generated new cluster ID: 2c8f2e174a684a4cb0da774754d202a7
I20260927 18:34:39.341120 11362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:39.351006 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.340620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.5.151.254:39139)
0927 18:34:39.340792 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.340801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.340846 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:34:39.340958 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.340965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.341023 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.341171 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.341872 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.343315 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.344235 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.344823 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.344936 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.347565 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:39.347569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:39.347570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:39.347580 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:39.349326 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.349718 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.349720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.349721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.349814 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.350040 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.350041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.350041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.350317 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:39.350444 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:39.350583 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:34:39.350704 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260927 18:34:39.357329 11362 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:39.357957 11362 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:39.360904 11362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d2d13f487c041eba41aaec170ac83df: Generated new TSK 0
I20260927 18:34:39.361085 11362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:39.383107 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39139
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:39.457406 11376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:39.457613 11376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:39.457648 11376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:39.458747 11376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:39.458801 11376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:39.458841 11376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:39.460157 11376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:39.460412 11376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:39.460616 11376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:39.465126 11381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:39.465097 11382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:39.465250 11376 server_base.cc:1034] running on GCE node
W20260927 18:34:39.465098 11384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:39.465564 11376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:39.465755 11376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:39.466883 11376 hybrid_clock.cc:648] HybridClock initialized: now 1790534079466870 us; error 26 us; skew 500 ppm
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.469146 11376 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:39.469489 11376 webserver.cc:460] Webserver started at http://127.5.151.193:45923/ using document root <none> and password file <none>
I20260927 18:34:39.469657 11376 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:39.469694 11376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:39.469771 11376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:39.470692 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "e4e56585c4d54ae685a789b8a60034ed"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "3bc439df16acdead5eafc49ef2da2605"
server_key_iv: "240ef60458959d459ad56089bb6ec2e6"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.470973 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "e4e56585c4d54ae685a789b8a60034ed"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "3bc439df16acdead5eafc49ef2da2605"
server_key_iv: "240ef60458959d459ad56089bb6ec2e6"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.472114 11376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:39.472744 11391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.472934 11376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:39.473035 11376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "e4e56585c4d54ae685a789b8a60034ed"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "3bc439df16acdead5eafc49ef2da2605"
server_key_iv: "240ef60458959d459ad56089bb6ec2e6"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.473119 11376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:39.482326 11376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:39.484045 11376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:39.484179 11376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:39.484375 11376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:39.484642 11376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:39.484671 11376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.484717 11376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:39.484741 11376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.489759 11376 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:38297
I20260927 18:34:39.489830 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:38297 every 8 connection(s)
I20260927 18:34:39.490082 11376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:39.497262 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11376
I20260927 18:34:39.497346 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:39.497545 5727 external_mini_cluster.cc:1428] Setting key 11ee13f53c86f4877485eeb4d8f00c2f
I20260927 18:34:39.498929 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.490797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:45807 (local address 127.5.151.254:39139)
0927 18:34:39.490860 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.490862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.490999 (+ 137us) server_negotiation.cc:407] Connection header received
0927 18:34:39.491854 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.491855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.491863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.491914 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.492458 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.492938 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.493832 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.494039 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.494088 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.496203 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:39.496204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:39.496205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:39.496211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:39.497403 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.497858 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.497859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.497860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.497923 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.498212 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.498213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.498213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.498359 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:39.498436 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:39.498700 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:34:39.498790 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:34:39.499018 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.490737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39139 (local address 127.5.151.193:45807)
0927 18:34:39.490887 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:39.490921 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:39.491774 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:39.491962 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:39.491965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:39.492195 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:39.492362 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:39.492366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.493020 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.493020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:39.493756 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:39.493758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.494119 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.494120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:39.494217 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.494738 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:39.494745 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:39.496116 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:39.497478 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:39.497485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:39.497487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:39.497784 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:39.497999 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:39.497999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:39.498000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:39.498122 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:39.498423 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:39.498428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:39.498532 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 18:34:39.498675 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 18:34:39.498822 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260927 18:34:39.500207 11505 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39139
I20260927 18:34:39.500298 11505 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:39.500473 11505 heartbeater.cc:511] Master 127.5.151.254:39139 requested a full tablet report, sending...
I20260927 18:34:39.501159 11314 ts_manager.cc:194] Registered new tserver with Master: e4e56585c4d54ae685a789b8a60034ed (127.5.151.193:38297)
I20260927 18:34:39.502048 11314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:45807
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:39.520907 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39139
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:39.595134 11510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:39.595321 11510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:39.595351 11510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:39.596369 11510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:39.596400 11510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:39.596423 11510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:39.597708 11510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:39.597865 11510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:39.598012 11510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:39.602327 11516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:39.602336 11518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:39.602331 11515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:39.602594 11510 server_base.cc:1034] running on GCE node
I20260927 18:34:39.602801 11510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:39.602995 11510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:39.604136 11510 hybrid_clock.cc:648] HybridClock initialized: now 1790534079604124 us; error 27 us; skew 500 ppm
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.606478 11510 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:39.606922 11510 webserver.cc:460] Webserver started at http://127.5.151.194:38107/ using document root <none> and password file <none>
I20260927 18:34:39.607155 11510 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:39.607219 11510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:39.607357 11510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:39.608366 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "773485c173324913921d49db573ebc44"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "c0615d08df0f7e852f488d91cc8962aa"
server_key_iv: "8aa59b2663784e6a609560ad6b515b9d"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.608733 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "773485c173324913921d49db573ebc44"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "c0615d08df0f7e852f488d91cc8962aa"
server_key_iv: "8aa59b2663784e6a609560ad6b515b9d"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.610055 11510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:39.610824 11525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.610989 11510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:39.611122 11510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "773485c173324913921d49db573ebc44"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "c0615d08df0f7e852f488d91cc8962aa"
server_key_iv: "8aa59b2663784e6a609560ad6b515b9d"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.611194 11510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:39.623773 11510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:39.625758 11510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:39.625907 11510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:39.626111 11510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:39.626397 11510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:39.626446 11510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.626477 11510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:39.626500 11510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.631875 11510 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:40003
I20260927 18:34:39.631915 11638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:40003 every 8 connection(s)
I20260927 18:34:39.632186 11510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:39.634932 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11510
I20260927 18:34:39.635010 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:39.635179 5727 external_mini_cluster.cc:1428] Setting key ea4b7722f52554af0562a7bbe6a34880
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:39.641611 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.632839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:57781 (local address 127.5.151.254:39139)
0927 18:34:39.632905 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.632906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.633100 (+ 194us) server_negotiation.cc:407] Connection header received
0927 18:34:39.634034 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.634036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.634048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.634105 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.634692 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.635308 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.636621 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.636787 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.636837 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.639002 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:39.639004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:39.639004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:39.639011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:39.640304 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.640693 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.640695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.640695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.640748 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.640996 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.640997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.640998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.641105 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:39.641176 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:39.641439 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 18:34:39.641518 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:34:39.641925 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.632838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39139 (local address 127.5.151.194:57781)
0927 18:34:39.632988 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:39.633022 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:39.633937 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:39.634177 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:39.634180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:39.634393 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:39.634602 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:39.634607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.635400 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.635401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:39.636539 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:39.636542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.636861 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.636862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:39.636992 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.637475 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:39.637484 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:39.638873 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:39.640393 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:39.640399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:39.640400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:39.640612 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:39.640808 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:39.640809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:39.640809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:39.640928 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:39.641180 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:39.641186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:39.641312 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 18:34:39.641409 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:34:39.641550 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260927 18:34:39.642405 11639 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39139
I20260927 18:34:39.642480 11639 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:39.642699 11639 heartbeater.cc:511] Master 127.5.151.254:39139 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
I20260927 18:34:39.643102 11314 ts_manager.cc:194] Registered new tserver with Master: 773485c173324913921d49db573ebc44 (127.5.151.194:40003)
I20260927 18:34:39.643471 11314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:57781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:39.658900 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39139
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:39.736181 11644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:39.736375 11644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:39.736397 11644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:39.737457 11644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:39.737488 11644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:39.737512 11644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:39.739063 11644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:39.739293 11644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:39.739470 11644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:39.744102 11650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:39.744118 11649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:39.744107 11652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:39.744374 11644 server_base.cc:1034] running on GCE node
I20260927 18:34:39.744576 11644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:39.744793 11644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:39.745960 11644 hybrid_clock.cc:648] HybridClock initialized: now 1790534079745945 us; error 30 us; skew 500 ppm
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.748383 11644 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:39.748780 11644 webserver.cc:460] Webserver started at http://127.5.151.195:35181/ using document root <none> and password file <none>
I20260927 18:34:39.749018 11644 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:39.749087 11644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:39.749230 11644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:39.750181 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "b26732c33a194bd89978338caeeca822"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "c3d008191fdb65308a254a9d15dc5102"
server_key_iv: "daa875fc4acdd5c6a670e3822c95c6b8"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.750527 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "b26732c33a194bd89978338caeeca822"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "c3d008191fdb65308a254a9d15dc5102"
server_key_iv: "daa875fc4acdd5c6a670e3822c95c6b8"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.751837 11644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:39.752511 11659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.752712 11644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:39.752807 11644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "b26732c33a194bd89978338caeeca822"
format_stamp: "Formatted at 2026-09-27 18:34:39 on dist-test-slave-19nl"
server_key: "c3d008191fdb65308a254a9d15dc5102"
server_key_iv: "daa875fc4acdd5c6a670e3822c95c6b8"
server_key_version: "encryptionkey@0"
I20260927 18:34:39.752880 11644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:39.785121 11644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:39.787101 11644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:39.787286 11644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:39.787540 11644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:39.787850 11644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:39.787902 11644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.787938 11644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:39.787983 11644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:39.793676 11644 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:33825
I20260927 18:34:39.793732 11772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:33825 every 8 connection(s)
I20260927 18:34:39.794071 11644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:39.802838 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.794835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:58717 (local address 127.5.151.254:39139)
0927 18:34:39.794906 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.794908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.795118 (+ 210us) server_negotiation.cc:407] Connection header received
0927 18:34:39.796033 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.796034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.796044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.796101 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.796652 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.797131 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.798039 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.798216 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.798262 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.800128 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:39.800130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:39.800130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:39.800138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:39.801454 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.801859 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.801860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.801861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.801935 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.802197 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.802198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.802198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.802306 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:39.802378 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:39.802672 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 18:34:39.802747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:34:39.803022 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.794814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39139 (local address 127.5.151.195:58717)
0927 18:34:39.794984 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:39.795025 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:39.795936 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:39.796154 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:39.796157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:39.796398 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:39.796579 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:39.796583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.797212 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.797213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:39.797969 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:39.797971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.798283 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.798284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:39.798380 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.798777 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:39.798785 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:39.800034 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:39.801551 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:39.801557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:39.801559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:39.801773 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:39.802002 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:39.802003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:39.802003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:39.802123 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:39.802370 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:39.802376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:39.802509 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 18:34:39.802656 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 18:34:39.802805 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260927 18:34:39.803375 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11644
I20260927 18:34:39.803478 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:39.803556 11773 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39139
I20260927 18:34:39.803668 11773 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:39.803683 5727 external_mini_cluster.cc:1428] Setting key e9fa223335f14f1aa00f60b73ff67b28
I20260927 18:34:39.803870 11773 heartbeater.cc:511] Master 127.5.151.254:39139 requested a full tablet report, sending...
I20260927 18:34:39.804258 11314 ts_manager.cc:194] Registered new tserver with Master: b26732c33a194bd89978338caeeca822 (127.5.151.195:33825)
I20260927 18:34:39.804680 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:34:39.804778 11314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:58717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/test-admin.keytab.
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.819379 5727 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 18:34:39.821048 11780 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:39.820326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34748 (local address 127.5.151.193:38297)
0927 18:34:39.820508 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.820512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.820522 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:39.820547 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.820549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.820562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.820627 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.820955 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34748: BlockingRecv error: recv got EOF from 127.0.0.1:34748 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
W20260927 18:34:39.821125 5727 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.822011 5727 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.822860 5727 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:39.830504 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.823346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.5.151.193:38297)
0927 18:34:39.823407 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.823409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.823463 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:34:39.823547 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.823548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.823559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.823609 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.823983 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.824681 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.825389 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.825704 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.825781 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.827741 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:39.827750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:39.827750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:39.827757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:39.829337 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.829717 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.829720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.829721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.829778 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.829969 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.829970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.829970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.830098 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:39.830235 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:39.830329 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:34:39.830398 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 18:34:39.832276 11780 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:39.831719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.5.151.193:38297)
0927 18:34:39.831774 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.831776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.831811 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:34:39.831904 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.831905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.831914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.831968 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.832222 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34768: BlockingRecv error: recv got EOF from 127.0.0.1:34768 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 18:34:39.832362 5727 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.833211 5727 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.834046 5727 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:39.840548 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.834492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.5.151.193:38297)
0927 18:34:39.834525 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.834526 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.834591 (+ 65us) server_negotiation.cc:407] Connection header received
0927 18:34:39.834695 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.834696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.834706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.834761 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.835139 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.835566 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.836223 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.836394 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.836439 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.838331 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:39.838332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:39.838332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:39.838338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:39.839634 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.839939 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.839939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.839940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.839983 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.840146 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.840146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.840147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.840224 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:39.840286 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:39.840383 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 18:34:39.840434 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 18:34:39.841763 11780 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:39.841367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.5.151.193:38297)
0927 18:34:39.841399 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.841401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.841431 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:34:39.841486 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.841486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.841493 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.841530 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.841737 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34782: BlockingRecv error: recv got EOF from 127.0.0.1:34782 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 18:34:39.841892 5727 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534055081918-5727-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.842792 5727 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:39.843576 5727 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:34:39.849807 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:39.844014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.5.151.193:38297)
0927 18:34:39.844066 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:39.844068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:39.844098 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:34:39.844163 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:39.844164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:39.844171 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:39.844235 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:39.844577 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.844990 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.845649 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:39.845825 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:39.845894 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:39.847889 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:39.847890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:39.847890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:39.847895 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:39.848926 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.849249 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.849250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.849250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.849294 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:39.849450 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:39.849450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:39.849451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:39.849527 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:39.849599 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:39.849703 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 18:34:39.849754 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:34:39.850502 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11376
I20260927 18:34:39.855225 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11510
I20260927 18:34:39.859485 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11644
I20260927 18:34:39.863476 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11282
2026-09-27T18:34:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1748 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11793](info): set up 2 sockets
Sep 27 18:34:39 dist-test-slave-19nl krb5kdc[11793](info): commencing operation
krb5kdc: starting...
W20260927 18:34:40.858712 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:40 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab.
Entry for principal oryx/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab.
I20260927 18:34:40.965384 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43035
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab
--principal=oryx/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:41.041344 11806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:41.041533 11806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:41.041555 11806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:41.042761 11806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:41.042802 11806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:41.042817 11806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:41.042830 11806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:41.042870 11806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:41.044281 11806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab
--principal=oryx/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:41.044510 11806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:41.044739 11806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:41.049209 11811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.049223 11812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.049430 11806 server_base.cc:1034] running on GCE node
W20260927 18:34:41.049257 11814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.049734 11806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:41.050015 11806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:41.051180 11806 hybrid_clock.cc:648] HybridClock initialized: now 1790534081051170 us; error 34 us; skew 500 ppm
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:41.054068 11806 init.cc:377] Logged in from keytab as oryx/127.5.151.254@KRBTEST.COM (short username oryx)
I20260927 18:34:41.054484 11806 webserver.cc:460] Webserver started at http://127.5.151.254:41435/ using document root <none> and password file <none>
I20260927 18:34:41.054736 11806 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.054806 11806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.054946 11806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:41.055940 11806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "9a85abd132ad4174a3206e8898af3222"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "3586b66bc3b6f513dd1919dc2c7b7d5e"
server_key_iv: "2ec85363981514b291978e40250c49e3"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.056298 11806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "9a85abd132ad4174a3206e8898af3222"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "3586b66bc3b6f513dd1919dc2c7b7d5e"
server_key_iv: "2ec85363981514b291978e40250c49e3"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.057564 11806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:41.058231 11821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.058403 11806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:41.058497 11806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "9a85abd132ad4174a3206e8898af3222"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "3586b66bc3b6f513dd1919dc2c7b7d5e"
server_key_iv: "2ec85363981514b291978e40250c49e3"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.058601 11806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.072541 11806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:41.074743 11806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:41.074901 11806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:41.078459 11806 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:43035
I20260927 18:34:41.078521 11873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:43035 every 8 connection(s)
I20260927 18:34:41.078871 11806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:41.079756 11874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:41.082038 11874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: Bootstrap starting.
I20260927 18:34:41.082605 11874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:41.082895 11874 log.cc:824] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:41.083544 11874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: No bootstrap required, opened a new log
I20260927 18:34:41.085069 11874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } }
I20260927 18:34:41.085196 11874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:41.085242 11874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a85abd132ad4174a3206e8898af3222, State: Initialized, Role: FOLLOWER
I20260927 18:34:41.085390 11874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } }
I20260927 18:34:41.085474 11874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:41.085528 11874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:41.085577 11874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:41.086354 11874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } }
I20260927 18:34:41.086462 11874 leader_election.cc:302] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a85abd132ad4174a3206e8898af3222; no voters:
I20260927 18:34:41.086638 11874 leader_election.cc:288] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:41.086669 11877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:41.086834 11877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 1 LEADER]: Becoming Leader. State: Replica: 9a85abd132ad4174a3206e8898af3222, State: Running, Role: LEADER
I20260927 18:34:41.086901 11874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:41.086972 11877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } }
I20260927 18:34:41.087360 11879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a85abd132ad4174a3206e8898af3222. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } }
I20260927 18:34:41.087440 11879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:41.087637 11878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } }
I20260927 18:34:41.087699 11878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:41.087973 11886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:41.088385 11886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:41.089114 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11806
I20260927 18:34:41.089223 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:41.089424 5727 external_mini_cluster.cc:1428] Setting key 1fac9c41e99cdf39f73333f606515774
I20260927 18:34:41.089781 11886 catalog_manager.cc:1347] Generated new cluster ID: 3e1305580d214a729e724d87943fe8a1
I20260927 18:34:41.089838 11886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:41.096951 11886 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:41.097407 11886 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:41.100752 11886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: Generated new TSK 0
I20260927 18:34:41.100901 11886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:34:41.101717 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.090048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.5.151.254:43035)
0927 18:34:41.090213 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.090222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.090854 (+ 632us) server_negotiation.cc:407] Connection header received
0927 18:34:41.090986 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.090992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.091044 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.091184 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.091931 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.093281 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.094350 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.095215 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.095315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.097993 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.097997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.097997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.098008 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.099945 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.100344 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.100347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.100348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.100451 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.100823 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.100824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.100824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.101099 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.101247 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.101340 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.101482 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab.
Entry for principal oryx/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab.
I20260927 18:34:41.125574 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43035
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab
--principal=oryx/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:41.202502 11900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:41.202751 11900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:41.202788 11900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:41.203930 11900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:41.203994 11900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:41.204036 11900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:41.205420 11900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab
--principal=oryx/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:41.205667 11900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:41.205886 11900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:41.211501 11908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.211511 11906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.211527 11905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.211930 11900 server_base.cc:1034] running on GCE node
I20260927 18:34:41.212152 11900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:41.212457 11900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:41.213608 11900 hybrid_clock.cc:648] HybridClock initialized: now 1790534081213597 us; error 25 us; skew 500 ppm
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:41.215920 11900 init.cc:377] Logged in from keytab as oryx/127.5.151.193@KRBTEST.COM (short username oryx)
I20260927 18:34:41.216368 11900 webserver.cc:460] Webserver started at http://127.5.151.193:36751/ using document root <none> and password file <none>
I20260927 18:34:41.216581 11900 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.216645 11900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.216795 11900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:41.217783 11900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "eb4d3886c1d544d984c4ba30ddb45751"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "06ae16729c56a55607a0c0af093e65f3"
server_key_iv: "c69032cc06e1032de99c6604079dc793"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.218123 11900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "eb4d3886c1d544d984c4ba30ddb45751"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "06ae16729c56a55607a0c0af093e65f3"
server_key_iv: "c69032cc06e1032de99c6604079dc793"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.219476 11900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:34:41.220124 11915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.220289 11900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:41.220381 11900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "eb4d3886c1d544d984c4ba30ddb45751"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "06ae16729c56a55607a0c0af093e65f3"
server_key_iv: "c69032cc06e1032de99c6604079dc793"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.220470 11900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.230072 11900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:41.231745 11900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:41.231886 11900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:41.232074 11900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:41.232339 11900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:41.232390 11900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.232427 11900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:41.232453 11900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.237757 11900 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:36311
I20260927 18:34:41.237794 12028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:36311 every 8 connection(s)
I20260927 18:34:41.238047 11900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:41.239845 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 11900
I20260927 18:34:41.239956 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:41.240132 5727 external_mini_cluster.cc:1428] Setting key 2c843c58b67c8f7c2d8aea8523144fd9
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:41.247206 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.238668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.193:53281)
0927 18:34:41.238815 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.238852 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.239287 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.239471 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.239474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.240244 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.240452 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.240458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.241131 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.241132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.241937 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.241939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.242320 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.242321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.242447 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.242959 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.242967 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.244244 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.245981 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.245987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.245988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.246185 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.246393 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.246393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.246394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.246480 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.246776 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.246780 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.246882 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.246962 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.247066 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260927 18:34:41.247205 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.238692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:53281 (local address 127.5.151.254:43035)
0927 18:34:41.238731 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.238733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.238933 (+ 200us) server_negotiation.cc:407] Connection header received
0927 18:34:41.239364 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.239364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.239373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.239425 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.240532 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.241048 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.242027 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.242231 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.242301 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.244329 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.244330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.244331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.244338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.245904 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.246257 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.246258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.246259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.246324 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.246541 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.246541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.246542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.246698 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.246782 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.247007 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.247095 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:34:41.247671 12029 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43035
I20260927 18:34:41.247780 12029 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:41.247968 12029 heartbeater.cc:511] Master 127.5.151.254:43035 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.5.151.194@KRBTEST.COM; defaulting to no policy
I20260927 18:34:41.248508 11838 ts_manager.cc:194] Registered new tserver with Master: eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193:36311)
I20260927 18:34:41.249075 11838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:53281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab.
Entry for principal oryx/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab.
I20260927 18:34:41.264876 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43035
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab
--principal=oryx/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:41.342896 12034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:41.343083 12034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:41.343106 12034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:41.344127 12034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:41.344161 12034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:41.344184 12034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:41.345477 12034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab
--principal=oryx/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:41.345643 12034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:41.345799 12034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:41.350473 12039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.350463 12042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.350466 12040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.350507 12034 server_base.cc:1034] running on GCE node
I20260927 18:34:41.350819 12034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:41.351037 12034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:41.352169 12034 hybrid_clock.cc:648] HybridClock initialized: now 1790534081352159 us; error 26 us; skew 500 ppm
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:41.354418 12034 init.cc:377] Logged in from keytab as oryx/127.5.151.194@KRBTEST.COM (short username oryx)
I20260927 18:34:41.354899 12034 webserver.cc:460] Webserver started at http://127.5.151.194:37193/ using document root <none> and password file <none>
I20260927 18:34:41.355114 12034 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.355156 12034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.355298 12034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:41.356230 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "de3327d4b00a4485b9f77b16b80b8365"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "108d3ed1654327f9cfd11c57bfc78073"
server_key_iv: "2fa1e311243952eefd83441b0a0f0f4b"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.356568 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "de3327d4b00a4485b9f77b16b80b8365"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "108d3ed1654327f9cfd11c57bfc78073"
server_key_iv: "2fa1e311243952eefd83441b0a0f0f4b"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.357764 12034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:41.358430 12049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.358621 12034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:41.358706 12034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "de3327d4b00a4485b9f77b16b80b8365"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "108d3ed1654327f9cfd11c57bfc78073"
server_key_iv: "2fa1e311243952eefd83441b0a0f0f4b"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.358758 12034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.371676 12034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:41.373775 12034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:41.373878 12034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:41.374035 12034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:41.374341 12034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:41.374379 12034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.374421 12034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:41.374446 12034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.379905 12034 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:36181
I20260927 18:34:41.379984 12162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:36181 every 8 connection(s)
I20260927 18:34:41.380273 12034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:41.388789 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.380918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:57723 (local address 127.5.151.254:43035)
0927 18:34:41.381004 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.381006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.381162 (+ 156us) server_negotiation.cc:407] Connection header received
0927 18:34:41.381603 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.381604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.381613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.381662 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.382751 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.383229 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.384180 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.384364 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.384414 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.386269 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.386270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.386270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.386276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.387650 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.387994 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.387995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.387995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.388041 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.388282 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.388283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.388283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.388378 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.388427 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.388638 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.388729 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:34:41.388991 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.380891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.194:57723)
0927 18:34:41.381041 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.381074 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.381519 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.381710 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.381713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.382344 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.382663 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.382668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.383326 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.383327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.384104 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.384107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.384434 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.384435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.384543 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.384929 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.384936 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.386185 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.387725 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.387732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.387738 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.387933 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.388123 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.388124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.388125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.388226 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.388433 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.388437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.388538 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.388614 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.388794 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260927 18:34:41.389058 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 12034
I20260927 18:34:41.389153 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:41.389333 5727 external_mini_cluster.cc:1428] Setting key 3aa714fb4f690dd3e5fb367d95edaa59
I20260927 18:34:41.389389 12163 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43035
I20260927 18:34:41.389478 12163 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:41.389659 12163 heartbeater.cc:511] Master 127.5.151.254:43035 requested a full tablet report, sending...
I20260927 18:34:41.390108 11838 ts_manager.cc:194] Registered new tserver with Master: de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181)
I20260927 18:34:41.390467 11838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:57723
WARNING: no policy specified for oryx/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab.
Entry for principal oryx/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab.
I20260927 18:34:41.411127 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43035
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab
--principal=oryx/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:41.487685 12168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:41.487879 12168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:41.487929 12168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:41.489166 12168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:41.489207 12168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:41.489233 12168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:41.490633 12168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab
--principal=oryx/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:41.490813 12168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:41.490980 12168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:41.495654 12173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.495658 12174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.495877 12176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.495862 12168 server_base.cc:1034] running on GCE node
I20260927 18:34:41.496119 12168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:41.496357 12168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:41.497504 12168 hybrid_clock.cc:648] HybridClock initialized: now 1790534081497495 us; error 40 us; skew 500 ppm
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:41.499938 12168 init.cc:377] Logged in from keytab as oryx/127.5.151.195@KRBTEST.COM (short username oryx)
I20260927 18:34:41.500314 12168 webserver.cc:460] Webserver started at http://127.5.151.195:35597/ using document root <none> and password file <none>
I20260927 18:34:41.500495 12168 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.500535 12168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.500622 12168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:41.501554 12168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "19f521890638442e9dee53f3f9156239"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "95cac4ef574dda873419d644cbe38709"
server_key_iv: "7e7b263d69d472acc50fb26b89b30a08"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.501832 12168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "19f521890638442e9dee53f3f9156239"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "95cac4ef574dda873419d644cbe38709"
server_key_iv: "7e7b263d69d472acc50fb26b89b30a08"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.503131 12168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:41.503783 12183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.503947 12168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:41.504042 12168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "19f521890638442e9dee53f3f9156239"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "95cac4ef574dda873419d644cbe38709"
server_key_iv: "7e7b263d69d472acc50fb26b89b30a08"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.504115 12168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.513319 12168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:41.515260 12168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:41.515412 12168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:41.515615 12168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:41.515903 12168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:41.515955 12168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.515996 12168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:41.516022 12168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.521575 12168 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:41229
I20260927 18:34:41.521610 12296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:41229 every 8 connection(s)
I20260927 18:34:41.521929 12168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:41.525429 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 12168
I20260927 18:34:41.525532 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:41.525689 5727 external_mini_cluster.cc:1428] Setting key bfe0eec57d67f0ad1e33fc6ee1c9ad23
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:41.530876 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.522614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:53681 (local address 127.5.151.254:43035)
0927 18:34:41.522673 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.522675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.522838 (+ 163us) server_negotiation.cc:407] Connection header received
0927 18:34:41.523398 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.523399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.523413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.523462 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.524387 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.524904 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.526026 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.526272 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.526354 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.528347 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.528348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.528348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.528355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.529729 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.530061 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.530061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.530062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.530105 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.530317 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.530317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.530317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.530414 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.530462 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.530754 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.530817 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:34:41.531069 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.522582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.195:53681)
0927 18:34:41.522728 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.522760 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.523287 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.523549 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.523553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.524103 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.524280 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.524284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.524997 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.524998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.525882 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.525885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.526375 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.526376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.526524 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.527015 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.527022 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.528268 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.529799 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.529803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.529804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.530003 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.530158 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.530158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.530159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.530246 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.530468 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.530472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.530623 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.530720 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.530843 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260927 18:34:41.531555 12297 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43035
I20260927 18:34:41.531635 12297 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:41.531857 12297 heartbeater.cc:511] Master 127.5.151.254:43035 requested a full tablet report, sending...
I20260927 18:34:41.532224 11838 ts_manager.cc:194] Registered new tserver with Master: 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:41.532577 11838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:53681
I20260927 18:34:41.537397 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.151.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.253.keytab.
Entry for principal oryx/127.5.151.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.253.keytab.
I20260927 18:34:41.560802 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:44143
--webserver_interface=127.5.151.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.253.keytab
--principal=oryx/127.5.151.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:43035,127.5.151.253:44143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:41.636276 12302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:41.636487 12302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:41.636519 12302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:41.637629 12302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:41.637694 12302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:41.637717 12302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:41.637737 12302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:41.637765 12302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:41.639146 12302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:43035,127.5.151.253:44143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.253.keytab
--principal=oryx/127.5.151.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:44143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.151.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
W20260927 18:34:41.639281 12302 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.151.254:43035,127.5.151.253:44143'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 18:34:41.639430 12302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:41.639643 12302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:41.644328 12302 server_base.cc:1034] running on GCE node
W20260927 18:34:41.644309 12310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.644297 12307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.644309 12308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.644721 12302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:41.645022 12302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:41.646203 12302 hybrid_clock.cc:648] HybridClock initialized: now 1790534081646174 us; error 57 us; skew 500 ppm
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:41.648579 12302 init.cc:377] Logged in from keytab as oryx/127.5.151.253@KRBTEST.COM (short username oryx)
I20260927 18:34:41.648984 12302 webserver.cc:460] Webserver started at http://127.5.151.253:43669/ using document root <none> and password file <none>
I20260927 18:34:41.649215 12302 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.649289 12302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.649437 12302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:41.650465 12302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/instance:
uuid: "0ff43d1f6acc40ef8b9c8469c74c641b"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "1fbb51166f081c13099257acccdc4225"
server_key_iv: "fd95127892b12b46e83434d1adff752e"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.650924 12302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal/instance:
uuid: "0ff43d1f6acc40ef8b9c8469c74c641b"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "1fbb51166f081c13099257acccdc4225"
server_key_iv: "fd95127892b12b46e83434d1adff752e"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.652197 12302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:41.652894 12317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.653067 12302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:41.653162 12302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
uuid: "0ff43d1f6acc40ef8b9c8469c74c641b"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "1fbb51166f081c13099257acccdc4225"
server_key_iv: "fd95127892b12b46e83434d1adff752e"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.653245 12302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.672312 12302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:41.673897 12302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:41.674041 12302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:41.677917 12302 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.253:44143
I20260927 18:34:41.677950 12369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.253:44143 every 8 connection(s)
I20260927 18:34:41.678284 12302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
I20260927 18:34:41.679191 12370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:41.680593 12370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:34:41.685101 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 12302
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.253@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:41.689622 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.681202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.5.151.254:43035)
0927 18:34:41.681277 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.681279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.681426 (+ 147us) server_negotiation.cc:407] Connection header received
0927 18:34:41.681977 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.681979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.681989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.682052 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.682742 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.683357 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.684290 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.684467 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.684517 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.686642 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.686643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.686644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.686651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.688097 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.688551 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.688551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.688552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.688618 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.688844 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.688845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.688845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.689027 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.689104 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.689431 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.689515 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:34:41.689761 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.681146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:37590)
0927 18:34:41.681310 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.681348 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.681853 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.682150 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.682154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.682401 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.682656 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.682661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.683432 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.683433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.684219 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.684221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.684534 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.684535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.684628 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.685069 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.685080 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.686532 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.688169 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.688175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.688177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.688479 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.688688 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.688688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.688689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.688784 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.689114 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.689119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.689280 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.689392 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.689520 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260927 18:34:41.690327 12370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.151.253@KRBTEST.COM
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.253@KRBTEST.COM for oryx/127.5.151.253@KRBTEST.COM
I20260927 18:34:41.694728 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.685585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.5.151.253:44143)
0927 18:34:41.685741 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.685747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.685771 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:34:41.685794 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.685796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.685815 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.685904 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.686494 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.687619 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.688637 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.689067 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.689129 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.691851 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.691857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.691857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.691862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.693503 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.693855 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.693858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.693859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.693920 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.694090 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.694091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.694091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.694261 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.694429 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.694502 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.694626 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260927 18:34:41.697261 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.690598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.0.0.1:39014)
0927 18:34:41.690629 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.690638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.690686 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.690909 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.690910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.691009 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.691053 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.691054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.691800 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.691801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.692389 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.692391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.692769 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.692769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.692836 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.693175 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.693179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.694378 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.696561 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.696561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.696562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.696713 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.696874 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.696875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.696875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.696920 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.697073 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.697074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.697101 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.697146 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.697210 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 18:34:41.697389 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.690657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.5.151.253:44143)
0927 18:34:41.690794 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.690796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.690802 (+ 6us) server_negotiation.cc:407] Connection header received
0927 18:34:41.690818 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.690819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.690827 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.690898 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.691125 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.691726 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.692480 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.692682 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.692768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.694470 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.694471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.694472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.694477 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.696497 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.696773 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.696774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.696774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.696821 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.696955 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.696956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.696956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.697033 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.697075 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.697159 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.697265 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260927 18:34:41.698776 12370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Bootstrap starting.
I20260927 18:34:41.699288 12370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:41.699502 12370 log.cc:824] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:41.700100 12370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: No bootstrap required, opened a new log
I20260927 18:34:41.701458 12370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } }
I20260927 18:34:41.701565 12370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:41.701586 12370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff43d1f6acc40ef8b9c8469c74c641b, State: Initialized, Role: FOLLOWER
I20260927 18:34:41.701694 12370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } }
I20260927 18:34:41.701913 12377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } } }
I20260927 18:34:41.701972 12370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:41.701984 12377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:34:41.703608 12388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:34:41.703670 12388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:34:41.705164 12302 master_runner.cc:428] Detected that this master 0ff43d1f6acc40ef8b9c8469c74c641b is joining an existing cluster
I20260927 18:34:41.705200 12302 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.151.253:44143
I20260927 18:34:41.711795 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.706083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.5.151.254:43035)
0927 18:34:41.706149 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.706151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.706465 (+ 314us) server_negotiation.cc:407] Connection header received
0927 18:34:41.706535 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.706536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.706543 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.706658 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.706960 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.707487 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.708244 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.708445 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.708518 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.709362 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.709364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.709364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.709371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.710690 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.711027 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.711028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.711028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.711078 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.711289 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.711290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.711290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.711394 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.711465 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.711620 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.711687 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:34:41.712059 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.706033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:37606)
0927 18:34:41.706405 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.706417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.706478 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.706663 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.706664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.706812 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.706875 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.706876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.707560 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.707562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.708129 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.708131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.708535 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.708536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.708594 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.708991 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.708994 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.709278 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.710784 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.710785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.710785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.710948 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.711145 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.711146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.711147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.711223 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.711457 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.711458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.711509 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.711594 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.711687 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":86,"threads_started":1}
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.254@KRBTEST.COM for oryx/127.5.151.253@KRBTEST.COM
I20260927 18:34:41.718883 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.706277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.5.151.253:44143)
0927 18:34:41.706307 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.706309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.712233 (+ 5924us) server_negotiation.cc:407] Connection header received
0927 18:34:41.712552 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.712553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.712564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.712628 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.712901 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.713546 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.714522 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.714832 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.714901 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.715766 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.715766 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.715767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.715773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.717646 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.718042 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.718043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.718043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.718099 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.718299 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.718300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.718300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.718441 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.718507 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.718717 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.718800 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 18:34:41.719125 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.706234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.0.0.1:39026)
0927 18:34:41.712171 (+ 5937us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.712182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.712254 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.712638 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.712639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.712773 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.712840 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.712842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.713614 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.713615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.714440 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.714442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.714908 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.714908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.714984 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.715366 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.715370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.715678 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.717739 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.717740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.717740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.717947 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.718167 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.718168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.718169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.718229 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.718511 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.718512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.718573 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.718947 (+ 374us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.719020 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5909,"thread_start_us":104,"threads_started":1}
I20260927 18:34:41.721529 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.714243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.5.151.253:44143)
0927 18:34:41.714278 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.714280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.714414 (+ 134us) server_negotiation.cc:407] Connection header received
0927 18:34:41.714477 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.714478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.714487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.714541 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.715071 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.715602 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.716665 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.716880 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.716958 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.719063 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.719065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.719065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.719072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.720473 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.720895 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.720896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.720896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.720939 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.721127 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.721128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.721128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.721212 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.721257 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.721430 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.721482 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:41.721621 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.714201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.0.0.1:39028)
0927 18:34:41.714304 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.714340 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.714412 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.714603 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.714608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.714931 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.714978 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.714980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.715696 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.715697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.716550 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.716553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.716962 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.716963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.717046 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.717437 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.717442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.718960 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.720550 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.720556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.720559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.720822 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.721002 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.721002 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.721003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.721069 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.721271 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.721277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.721325 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.721416 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.721523 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":51,"threads_started":1}
I20260927 18:34:41.722084 11838 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.151.253:44143: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b"
member_type: NON_VOTER
last_known_addr {
host: "127.5.151.253"
port: 44143
}
attrs {
promote: true
}
}
dest_uuid: "9a85abd132ad4174a3206e8898af3222"
cas_config_opid_index: -1
I20260927 18:34:41.722378 11838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } }
I20260927 18:34:41.722865 12398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0ff43d1f6acc40ef8b9c8469c74c641b (127.5.151.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } }
I20260927 18:34:41.723138 12344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:41.723774 12398 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.151.253:44143
I20260927 18:34:41.723954 12302 master.cc:537] Master@127.5.151.253:44143 shutting down...
I20260927 18:34:41.724042 12344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Refusing update from remote peer 9a85abd132ad4174a3206e8898af3222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 18:34:41.723954 12397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } }
I20260927 18:34:41.724383 12397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:41.724468 12398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:34:41.725302 11823 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 18:34:41.725734 11823 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 -> Peer 0ff43d1f6acc40ef8b9c8469c74c641b (127.5.151.253:44143): Couldn't send request to peer 0ff43d1f6acc40ef8b9c8469c74c641b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:34:41.726688 12302 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:41.726778 12302 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:41.726826 12302 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: stopping tablet replica
I20260927 18:34:41.739087 12302 master.cc:552] Master@127.5.151.253:44143 shutdown complete.
I20260927 18:34:41.739132 12302 master_runner.cc:305] Clearing existing system tablet
I20260927 18:34:41.739250 12302 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.739295 12302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.739909 12302 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:41.740356 12402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.740486 12302 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.740541 12302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.740563 12302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
uuid: "0ff43d1f6acc40ef8b9c8469c74c641b"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "1fbb51166f081c13099257acccdc4225"
server_key_iv: "fd95127892b12b46e83434d1adff752e"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.741277 12302 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:41.743047 12302 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:34:41.743116 12302 log.cc:1196] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 18:34:41.743321 12302 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Deleting consensus metadata
I20260927 18:34:41.743428 12302 master_runner.cc:315] Copying system tablet from 127.5.151.254:43035
I20260927 18:34:41.744014 12302 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.151.254:43035
I20260927 18:34:41.750033 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.744357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.5.151.254:43035)
0927 18:34:41.744397 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.744399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.744494 (+ 95us) server_negotiation.cc:407] Connection header received
0927 18:34:41.744527 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.744528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.744538 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.744610 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.744870 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.745610 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.746360 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.746678 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.746766 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.747690 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.747691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.747691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.747698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.748952 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.749301 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.749302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.749302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.749356 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.749561 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.749562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.749563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.749666 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.749738 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.749878 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.749947 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:41.750051 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.744290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:37612)
0927 18:34:41.744432 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.744441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.744491 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.744612 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.744612 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.744748 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.744807 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.744809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.745695 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.745696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.746282 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.746284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.746766 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.746767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.746814 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.747277 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.747282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.747609 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.749051 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.749052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.749052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.749214 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.749425 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.749425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.749426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.749499 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.749729 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.749730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.749771 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.749855 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.749948 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260927 18:34:41.750301 11858 tablet_copy_service.cc:140] P 9a85abd132ad4174a3206e8898af3222: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ff43d1f6acc40ef8b9c8469c74c641b ({username='oryx', principal='oryx/127.5.151.253@KRBTEST.COM'} at 127.0.0.1:37612)
I20260927 18:34:41.750392 11858 tablet_copy_service.cc:161] P 9a85abd132ad4174a3206e8898af3222: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ff43d1f6acc40ef8b9c8469c74c641b at {username='oryx', principal='oryx/127.5.151.253@KRBTEST.COM'} at 127.0.0.1:37612: session id = 0ff43d1f6acc40ef8b9c8469c74c641b-00000000000000000000000000000000
I20260927 18:34:41.751082 11858 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:34:41.751698 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:34:41.752830 12302 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: tablet copy: Starting download of 0 data blocks...
I20260927 18:34:41.752969 12302 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: tablet copy: Starting download of 1 WAL segments...
I20260927 18:34:41.753760 12302 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:34:41.754576 11858 tablet_copy_service.cc:342] P 9a85abd132ad4174a3206e8898af3222: Request end of tablet copy session 0ff43d1f6acc40ef8b9c8469c74c641b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.151.253@KRBTEST.COM'} at 127.0.0.1:37612
I20260927 18:34:41.754628 11858 tablet_copy_service.cc:433] P 9a85abd132ad4174a3206e8898af3222: ending tablet copy session 0ff43d1f6acc40ef8b9c8469c74c641b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ff43d1f6acc40ef8b9c8469c74c641b
I20260927 18:34:41.756914 12302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:41.758059 12412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.758064 12409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.758248 12302 server_base.cc:1034] running on GCE node
W20260927 18:34:41.758692 12410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.758829 12302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:41.758951 12302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:41.760088 12302 hybrid_clock.cc:648] HybridClock initialized: now 1790534081760081 us; error 31 us; skew 500 ppm
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:41.765265 12302 init.cc:377] Logged in from keytab as oryx/127.5.151.253@KRBTEST.COM (short username oryx)
I20260927 18:34:41.765534 12302 webserver.cc:460] Webserver started at http://127.5.151.253:39851/ using document root <none> and password file <none>
I20260927 18:34:41.765645 12302 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.765694 12302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.766234 12302 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.766778 12419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.766921 12302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.766976 12302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
uuid: "0ff43d1f6acc40ef8b9c8469c74c641b"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "1fbb51166f081c13099257acccdc4225"
server_key_iv: "fd95127892b12b46e83434d1adff752e"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.767030 12302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.779946 12302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:41.781440 12302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:41.784333 12302 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.253:44143
I20260927 18:34:41.784474 12471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.253:44143 every 8 connection(s)
I20260927 18:34:41.784662 12302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
I20260927 18:34:41.785285 12472 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:34:41.785643 12472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Bootstrap starting.
I20260927 18:34:41.787856 12472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:34:41.788093 12472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: Bootstrap complete.
I20260927 18:34:41.788288 12472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } }
I20260927 18:34:41.788350 12472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ff43d1f6acc40ef8b9c8469c74c641b, State: Initialized, Role: LEARNER
I20260927 18:34:41.788374 12472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } }
I20260927 18:34:41.788538 12472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:41.788553 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } }
I20260927 18:34:41.788736 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: LEARNER
I20260927 18:34:41.789937 12486 catalog_manager.cc:1259] Loaded cluster ID: 3e1305580d214a729e724d87943fe8a1
I20260927 18:34:41.789994 12486 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: loading cluster ID for follower catalog manager: success
I20260927 18:34:41.790961 12486 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: acquiring CA information for follower catalog manager: success
I20260927 18:34:41.791232 12486 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:34:41.802385 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.796221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.5.151.253:44143)
0927 18:34:41.796334 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.796336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.796344 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:41.796368 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.796368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.796382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.796458 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.796957 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.797642 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.798346 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.798628 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.798709 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.799853 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.799854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.799854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.799861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.801247 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.801619 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.801620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.801621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.801682 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.801881 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.801882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.801883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.801989 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.802088 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.802190 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.802276 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
WARNING: no policy specified for oryx/127.5.151.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.252.keytab.
Entry for principal oryx/127.5.151.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.252.keytab.
I20260927 18:34:41.824795 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:44941
--webserver_interface=127.5.151.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.252.keytab
--principal=oryx/127.5.151.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:41.903317 12490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:41.903499 12490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:41.903520 12490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:41.904670 12490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:41.904711 12490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:41.904726 12490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:41.904783 12490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:41.904809 12490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:41.906234 12490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.252.keytab
--principal=oryx/127.5.151.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:44941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.151.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:41.906464 12490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:41.906747 12490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:41.911405 12490 server_base.cc:1034] running on GCE node
W20260927 18:34:41.911377 12498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.911381 12495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:41.911378 12496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:41.911729 12490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:41.911993 12490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:41.913160 12490 hybrid_clock.cc:648] HybridClock initialized: now 1790534081913139 us; error 34 us; skew 500 ppm
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:41.915571 12490 init.cc:377] Logged in from keytab as oryx/127.5.151.252@KRBTEST.COM (short username oryx)
I20260927 18:34:41.916014 12490 webserver.cc:460] Webserver started at http://127.5.151.252:34251/ using document root <none> and password file <none>
I20260927 18:34:41.916244 12490 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:41.916318 12490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:41.916465 12490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:41.917461 12490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/instance:
uuid: "b2c6219c34b74bcd974ff88591ab1ca3"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "4a8c042cc924590bf6a7de0e8a003f95"
server_key_iv: "f57333aa90365af5b181859dad02de35"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.917809 12490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal/instance:
uuid: "b2c6219c34b74bcd974ff88591ab1ca3"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "4a8c042cc924590bf6a7de0e8a003f95"
server_key_iv: "f57333aa90365af5b181859dad02de35"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.919219 12490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 18:34:41.919986 12505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:41.920151 12490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:41.920240 12490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
uuid: "b2c6219c34b74bcd974ff88591ab1ca3"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "4a8c042cc924590bf6a7de0e8a003f95"
server_key_iv: "f57333aa90365af5b181859dad02de35"
server_key_version: "encryptionkey@0"
I20260927 18:34:41.920320 12490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:41.934372 12490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:41.936185 12490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:41.936334 12490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:41.940089 12490 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.252:44941
I20260927 18:34:41.940117 12557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.252:44941 every 8 connection(s)
I20260927 18:34:41.940439 12490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
I20260927 18:34:41.941310 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:41.942775 12558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.252@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:41.949043 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 12490
I20260927 18:34:41.950708 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.943490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.5.151.254:43035)
0927 18:34:41.943564 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.943566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.943606 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:34:41.944031 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.944032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.944040 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.944093 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.944601 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.945081 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.945963 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.946134 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.946181 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.948059 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.948061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.948061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.948067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.949254 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.949733 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.949734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.949735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.949798 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.950065 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.950066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.950066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.950173 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.950242 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.950533 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.950647 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:34:41.950870 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.943350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:37620)
0927 18:34:41.943501 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.943538 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.943953 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.944141 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.944144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.944361 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.944537 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.944540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.945149 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.945150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.945892 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.945894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.946208 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.946209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.946303 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.946742 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.946749 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.947979 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.949348 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.949354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.949356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.949636 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.949876 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.949877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.949877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.949989 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.950244 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.950250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.950382 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.950495 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.950670 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260927 18:34:41.951392 12558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.151.252@KRBTEST.COM
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.252@KRBTEST.COM for oryx/127.5.151.253@KRBTEST.COM
I20260927 18:34:41.958088 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.949519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.5.151.252:44941)
0927 18:34:41.949664 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.949668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.949715 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:34:41.949765 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.949767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.949791 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.949866 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.950466 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.951634 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.952784 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.953183 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.953254 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.955565 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.955571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.955572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.955578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.956933 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.957299 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.957302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.957302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.957358 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.957555 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.957556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.957556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.957692 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.957842 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.957911 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.957982 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260927 18:34:41.958681 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.951638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.0.0.1:39064)
0927 18:34:41.951684 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.951694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.951728 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.951863 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.951863 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.951964 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.952003 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.952004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.952894 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.952895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.953539 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.953541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.953909 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.953910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.953973 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.954348 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.954352 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.955977 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.957586 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.957587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.957587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.957752 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.957948 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.957948 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.957949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.958004 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.958385 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.958386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.958419 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.958486 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.958598 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:34:41.958716 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.951696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.5.151.253:44143)
0927 18:34:41.951746 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.951748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.951756 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:41.951785 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.951786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.951797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.951859 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.952070 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.952812 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.953620 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.953838 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.953911 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.956066 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.956067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.956067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.956073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.957490 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.957823 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.957824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.957825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.957879 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.958163 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.958164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.958164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.958297 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.958354 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.958526 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.958626 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:34:41.959012 12558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.252@KRBTEST.COM for oryx/127.5.151.252@KRBTEST.COM
I20260927 18:34:41.964629 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.959224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.0.0.1:37222)
0927 18:34:41.959267 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.959277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.959312 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.959437 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.959437 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.959557 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.959577 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.959578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.960167 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.960168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.960689 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.960690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.961048 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.961049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.961095 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.961479 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.961483 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.962711 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.963883 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.963883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.963884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.964034 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.964189 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.964190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.964190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.964235 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.964419 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.964420 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.964451 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.964496 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.964560 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 18:34:41.964726 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.959279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.5.151.252:44941)
0927 18:34:41.959309 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.959311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.959319 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:41.959367 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.959368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.959377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.959434 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.959649 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.960081 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.960770 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.960971 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.961048 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.962778 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.962778 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.962779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.962784 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.963815 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.964093 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.964094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.964094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.964137 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.964291 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.964292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.964292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.964366 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.964411 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.964511 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.964601 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":16}
I20260927 18:34:41.965857 12558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Bootstrap starting.
I20260927 18:34:41.966681 12558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:41.966993 12558 log.cc:824] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:41.967670 12558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: No bootstrap required, opened a new log
I20260927 18:34:41.969123 12558 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } }
I20260927 18:34:41.969234 12558 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:41.969256 12558 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c6219c34b74bcd974ff88591ab1ca3, State: Initialized, Role: FOLLOWER
I20260927 18:34:41.969337 12558 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } }
I20260927 18:34:41.969552 12563 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } } }
I20260927 18:34:41.969611 12558 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:41.969651 12563 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:34:41.971174 12574 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:34:41.971246 12574 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:34:41.973335 12490 master_runner.cc:428] Detected that this master b2c6219c34b74bcd974ff88591ab1ca3 is joining an existing cluster
I20260927 18:34:41.973403 12490 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.151.252:44941
I20260927 18:34:41.980909 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.974408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.5.151.252:44941)
0927 18:34:41.974446 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.974448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.974619 (+ 171us) server_negotiation.cc:407] Connection header received
0927 18:34:41.974711 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.974712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.974720 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.974747 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.975207 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.975917 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.976988 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.977253 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.977288 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.978307 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.978308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.978308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.978315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.979667 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.980099 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.980100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.980101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.980158 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.980349 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.980350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.980350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.980488 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.980531 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.980727 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.980805 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:41.981237 12580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.974373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.0.0.1:37230)
0927 18:34:41.974531 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.974539 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.974642 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.974802 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.974803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.975053 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.975124 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.975126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.975998 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.975999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.976904 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.976905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.977367 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.977367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.977424 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.977885 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.977890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.978203 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.979762 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.979763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.979763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.979932 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.980228 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.980228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.980229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.980283 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.980573 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.980574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.980611 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.981019 (+ 408us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.981117 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":37,"thread_start_us":86,"threads_started":1}
I20260927 18:34:41.998458 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.974415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.5.151.254:43035)
0927 18:34:41.974464 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.974466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.974696 (+ 230us) server_negotiation.cc:407] Connection header received
0927 18:34:41.974794 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.974795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.974805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.974835 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.975246 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.975965 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.976993 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.977283 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.977319 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.995506 (+ 18187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.995508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.995508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.995516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.997175 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997614 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.997615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.997615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.997672 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997886 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.997887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.997887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.998027 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.998093 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.998289 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.998363 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 18:34:41.998471 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.974512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:37626)
0927 18:34:41.974640 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.974650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.974719 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.974895 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.974896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.975096 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.975163 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.975164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.976047 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.976047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.976900 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.976902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.977385 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.977386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.977452 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.995067 (+ 17615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.995072 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.995410 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.997264 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.997266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.997536 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.997744 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.997745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.997822 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.998114 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.998114 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.998159 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.998255 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.998358 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260927 18:34:41.998579 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.974736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.5.151.253:44143)
0927 18:34:41.974776 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:41.974778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:41.974980 (+ 202us) server_negotiation.cc:407] Connection header received
0927 18:34:41.975070 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:41.975071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:41.975081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:41.975108 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:41.975536 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.976255 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.977285 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.977573 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.977613 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.995669 (+ 18056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:41.995671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:41.995671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:41.995678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:41.997298 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997706 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.997707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.997707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.997766 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997959 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:41.997960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:41.997961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:41.998119 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:41.998194 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:41.998376 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 18:34:41.998462 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:41.999338 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:41.974814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.0.0.1:39068)
0927 18:34:41.974925 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:41.974934 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:41.975002 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:41.975173 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:41.975173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:41.975380 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:41.975449 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.975450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.976365 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.976366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.977179 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:41.977182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:41.977687 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:41.977688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:41.977750 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:41.995147 (+ 17397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:41.995152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:41.995578 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:41.997383 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.997384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.997631 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.997834 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:41.997835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:41.997835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:41.997890 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:41.998197 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:41.998198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:41.998242 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 18:34:41.998960 (+ 718us) client_negotiation.cc:240] Negotiation successful
0927 18:34:41.999061 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
Sep 27 18:34:41 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534081, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.254@KRBTEST.COM for oryx/127.5.151.252@KRBTEST.COM
I20260927 18:34:42.006426 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.000534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.5.151.252:44941)
0927 18:34:42.000572 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.000574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.000582 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:42.000632 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.000632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.000641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.000695 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.000957 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.001399 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.002078 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.002277 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.002358 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.004117 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.004118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.004118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.004125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.005350 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.005679 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.005680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.005680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.005732 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.005931 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.005932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.005932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.006049 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.006119 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.006284 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.006342 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:42.006440 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.000477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.0.0.1:37232)
0927 18:34:42.000523 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.000532 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.000567 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.000700 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.000701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.000842 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.000878 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.000880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.001484 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.001485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.001988 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.001989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.002358 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.002359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.002429 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.002807 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.002811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.004020 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.005434 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.005435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.005436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.005601 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.005805 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.005805 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.005806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.005867 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.006112 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.006113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.006153 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.006243 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.006342 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 18:34:42.006903 11838 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.151.252:44941: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3"
member_type: NON_VOTER
last_known_addr {
host: "127.5.151.252"
port: 44941
}
attrs {
promote: true
}
}
dest_uuid: "9a85abd132ad4174a3206e8898af3222"
cas_config_opid_index: 5
I20260927 18:34:42.007149 11838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } }
I20260927 18:34:42.007507 12398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.007809 12532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:42.008419 12398 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.151.252:44941
I20260927 18:34:42.008486 12398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.008543 12398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:42.008608 12532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Refusing update from remote peer 9a85abd132ad4174a3206e8898af3222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:34:42.008828 12490 master.cc:537] Master@127.5.151.252:44941 shutting down...
I20260927 18:34:42.008929 12397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:34:42.009536 11824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 -> Peer b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252:44941): Couldn't send request to peer b2c6219c34b74bcd974ff88591ab1ca3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:34:42.012475 12490 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:42.012554 12490 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:42.012564 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.007562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.0.0.1:39080)
0927 18:34:42.007591 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.007602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.007638 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.009274 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.009275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.009276 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.009310 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.009311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.010197 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.010198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.011465 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.011468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.012299 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.012300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.012363 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.012383 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.012443 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.012483 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 18:34:42.012595 12490 tablet_replica.cc:330] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: stopping tablet replica
I20260927 18:34:42.012811 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.008774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.5.151.253:44143)
0927 18:34:42.009088 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.009089 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.009097 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:42.009116 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.009119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.009141 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.009217 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.009381 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.010112 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.011548 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.012136 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.012193 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.012267 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.012687 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.012723 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292}
I20260927 18:34:42.012984 12446 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 LEARNER]: Refusing update from remote peer 9a85abd132ad4174a3206e8898af3222: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:34:42.013159 12397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 18:34:42.013720 12446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.014998 12584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.015090 12584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: LEARNER
I20260927 18:34:42.015295 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a85abd132ad4174a3206e8898af3222. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: NON_VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: true } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.015342 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: LEARNER
I20260927 18:34:42.015823 12397 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: attempting to promote NON_VOTER 0ff43d1f6acc40ef8b9c8469c74c641b to VOTER
I20260927 18:34:42.015972 12397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } }
I20260927 18:34:42.016254 12490 master.cc:552] Master@127.5.151.252:44941 shutdown complete.
I20260927 18:34:42.016296 12490 master_runner.cc:305] Clearing existing system tablet
I20260927 18:34:42.016420 12490 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:42.016417 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.016319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.0.0.1:37236)
0927 18:34:42.016346 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.016383 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:44941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260927 18:34:42.016458 12490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:42.016469 12446 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 LEARNER]: Refusing update from remote peer 9a85abd132ad4174a3206e8898af3222: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 18:34:42.016667 12585 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 18:34:42.016748 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.016680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.0.0.1:37246)
0927 18:34:42.016710 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.016731 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:44941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 18:34:42.016798 11824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 -> Peer b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252:44941): Couldn't send request to peer b2c6219c34b74bcd974ff88591ab1ca3. Status: Network error: Client connection negotiation failed: client connection to 127.5.151.252:44941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:34:42.017163 12490 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:42.017266 12398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0ff43d1f6acc40ef8b9c8469c74c641b (127.5.151.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.017405 12446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0ff43d1f6acc40ef8b9c8469c74c641b (127.5.151.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.018206 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.018211 12398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.018283 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:34:42.018242 12588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.018283 12398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:42.018472 12588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:42.018657 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.018759 12490 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:42.018803 12490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 18:34:42.018826 12490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
uuid: "b2c6219c34b74bcd974ff88591ab1ca3"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "4a8c042cc924590bf6a7de0e8a003f95"
server_key_iv: "f57333aa90365af5b181859dad02de35"
server_key_version: "encryptionkey@0"
I20260927 18:34:42.019582 12490 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:42.020875 12490 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:34:42.020943 12490 log.cc:1196] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 18:34:42.021152 12490 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Deleting consensus metadata
I20260927 18:34:42.021333 12490 master_runner.cc:315] Copying system tablet from 127.5.151.254:43035
I20260927 18:34:42.021912 12490 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: tablet copy: Beginning tablet copy session from remote peer at address 127.5.151.254:43035
I20260927 18:34:42.027917 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.022159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:37642)
0927 18:34:42.022277 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.022288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.022340 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.022482 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.022483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.022681 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.022724 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.022726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.023572 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.023572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.024185 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.024187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.024649 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.024650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.024695 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.025124 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.025128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.025391 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.027044 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.027045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.027045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.027204 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.027408 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.027409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.027409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.027456 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.027702 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.027703 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.027734 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.027781 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.027848 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":98,"threads_started":1}
I20260927 18:34:42.027956 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.022235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.5.151.254:43035)
0927 18:34:42.022280 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.022281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.022329 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:34:42.022403 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.022404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.022412 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.022462 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.022789 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.023504 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.024264 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.024570 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.024646 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.025478 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.025479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.025479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.025486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.026954 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.027280 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.027281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.027281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.027340 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.027513 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.027514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.027514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.027644 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.027709 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.027803 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.027872 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:42.028285 11858 tablet_copy_service.cc:140] P 9a85abd132ad4174a3206e8898af3222: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b2c6219c34b74bcd974ff88591ab1ca3 ({username='oryx', principal='oryx/127.5.151.252@KRBTEST.COM'} at 127.0.0.1:37642)
I20260927 18:34:42.028342 11858 tablet_copy_service.cc:161] P 9a85abd132ad4174a3206e8898af3222: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b2c6219c34b74bcd974ff88591ab1ca3 at {username='oryx', principal='oryx/127.5.151.252@KRBTEST.COM'} at 127.0.0.1:37642: session id = b2c6219c34b74bcd974ff88591ab1ca3-00000000000000000000000000000000
I20260927 18:34:42.028723 11858 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:34:42.029127 12490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:34:42.029954 12490 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: tablet copy: Starting download of 0 data blocks...
I20260927 18:34:42.030088 12490 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: tablet copy: Starting download of 1 WAL segments...
I20260927 18:34:42.030844 12490 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:34:42.031669 11858 tablet_copy_service.cc:342] P 9a85abd132ad4174a3206e8898af3222: Request end of tablet copy session b2c6219c34b74bcd974ff88591ab1ca3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.151.252@KRBTEST.COM'} at 127.0.0.1:37642
I20260927 18:34:42.031718 11858 tablet_copy_service.cc:433] P 9a85abd132ad4174a3206e8898af3222: ending tablet copy session b2c6219c34b74bcd974ff88591ab1ca3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b2c6219c34b74bcd974ff88591ab1ca3
I20260927 18:34:42.033855 12490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:42.034900 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:42.034960 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:42.035302 12598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:42.035434 12490 server_base.cc:1034] running on GCE node
I20260927 18:34:42.035553 12490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:42.035693 12490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:42.036803 12490 hybrid_clock.cc:648] HybridClock initialized: now 1790534082036795 us; error 23 us; skew 500 ppm
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:42.038187 12490 init.cc:377] Logged in from keytab as oryx/127.5.151.252@KRBTEST.COM (short username oryx)
I20260927 18:34:42.038429 12490 webserver.cc:460] Webserver started at http://127.5.151.252:33511/ using document root <none> and password file <none>
I20260927 18:34:42.038573 12490 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:42.038630 12490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:42.039191 12490 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.039695 12605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.039855 12490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.039927 12490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
uuid: "b2c6219c34b74bcd974ff88591ab1ca3"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "4a8c042cc924590bf6a7de0e8a003f95"
server_key_iv: "f57333aa90365af5b181859dad02de35"
server_key_version: "encryptionkey@0"
I20260927 18:34:42.039981 12490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:42.063220 12490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:42.064916 12490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:42.067950 12490 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.252:44941
I20260927 18:34:42.067966 12657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.252:44941 every 8 connection(s)
I20260927 18:34:42.068288 12490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
I20260927 18:34:42.068940 12658 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:34:42.069307 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Bootstrap starting.
I20260927 18:34:42.071504 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:34:42.071703 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Bootstrap complete.
I20260927 18:34:42.071879 12658 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } }
I20260927 18:34:42.071965 12658 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2c6219c34b74bcd974ff88591ab1ca3, State: Initialized, Role: LEARNER
I20260927 18:34:42.072013 12658 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } }
I20260927 18:34:42.072181 12658 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:42.072160 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.072225 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:34:42.073783 12672 catalog_manager.cc:1259] Loaded cluster ID: 3e1305580d214a729e724d87943fe8a1
I20260927 18:34:42.073835 12672 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: loading cluster ID for follower catalog manager: success
I20260927 18:34:42.075222 12672 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: acquiring CA information for follower catalog manager: success
I20260927 18:34:42.075500 12672 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:34:42.121367 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.115670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.5.151.252:44941)
0927 18:34:42.115791 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.115792 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.115801 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:42.115825 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.115826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.115834 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.115893 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.116446 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.117060 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.117727 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.117901 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.117956 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.119082 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.119083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.119083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.119089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.120512 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.120784 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.120785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.120785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.120833 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.120970 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.120970 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.120970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.121054 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.121119 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.121210 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.121264 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260927 18:34:42.121924 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11900
I20260927 18:34:42.126407 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:36311
--local_ip_for_outbound_sockets=127.5.151.193
--tserver_master_addrs=127.5.151.254:43035
--webserver_port=36751
--webserver_interface=127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab
--principal=oryx/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:42.204681 12674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:42.204865 12674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:42.204887 12674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:42.205916 12674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:42.205947 12674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:42.205971 12674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:42.207317 12674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab
--principal=oryx/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:36311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=36751
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:42.207494 12674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:42.207650 12674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:42.212376 12680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:42.212399 12679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:42.212406 12674 server_base.cc:1034] running on GCE node
W20260927 18:34:42.212389 12682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:42.212719 12674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:42.212982 12674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:42.214138 12674 hybrid_clock.cc:648] HybridClock initialized: now 1790534082214131 us; error 34 us; skew 500 ppm
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:42.216648 12674 init.cc:377] Logged in from keytab as oryx/127.5.151.193@KRBTEST.COM (short username oryx)
I20260927 18:34:42.217056 12674 webserver.cc:460] Webserver started at http://127.5.151.193:36751/ using document root <none> and password file <none>
I20260927 18:34:42.217273 12674 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:42.217353 12674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:42.218710 12674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:42.219431 12689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.219632 12674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:42.219718 12674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "eb4d3886c1d544d984c4ba30ddb45751"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "06ae16729c56a55607a0c0af093e65f3"
server_key_iv: "c69032cc06e1032de99c6604079dc793"
server_key_version: "encryptionkey@0"
I20260927 18:34:42.220039 12674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:42.229270 12674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:42.231498 12674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:42.231647 12674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:42.231851 12674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:42.232131 12674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:42.232163 12674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.232182 12674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:42.232227 12674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.238111 12674 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:36311
I20260927 18:34:42.238188 12802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:36311 every 8 connection(s)
I20260927 18:34:42.238538 12674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:42.240700 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 12674
I20260927 18:34:42.240862 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 12034
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.253@KRBTEST.COM
I20260927 18:34:42.247359 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:36181
--local_ip_for_outbound_sockets=127.5.151.194
--tserver_master_addrs=127.5.151.254:43035
--webserver_port=37193
--webserver_interface=127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab
--principal=oryx/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.252@KRBTEST.COM
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:42.251492 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.239272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:45443 (local address 127.5.151.252:44941)
0927 18:34:42.239334 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.239337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.239542 (+ 205us) server_negotiation.cc:407] Connection header received
0927 18:34:42.240650 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.240652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.240667 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.240696 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.242250 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.243154 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.244665 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.244977 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.245013 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.248216 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.248217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.248218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.248224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.249886 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250279 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.250280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.250281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.250351 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250635 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.250636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.250636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.250793 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.250856 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.251230 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.251349 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:34:42.251824 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.239255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:37367 (local address 127.5.151.253:44143)
0927 18:34:42.239302 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.239304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.239519 (+ 215us) server_negotiation.cc:407] Connection header received
0927 18:34:42.240648 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.240649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.240663 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.240696 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.242242 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.243062 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.244650 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.244979 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.245017 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.247875 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.247876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.247877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.247883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.250464 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250812 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.250813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.250813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.250868 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.251061 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.251062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.251062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.251181 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.251275 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.251660 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.251733 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:42.251830 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.239711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:47655 (local address 127.5.151.254:43035)
0927 18:34:42.239762 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.239765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.239775 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:42.240648 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.240649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.240662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.240696 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.242250 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.243142 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.244665 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.244946 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.244985 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.248901 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.248902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.248902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.248910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.250626 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250989 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.250991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.250991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.251045 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.251281 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.251282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.251283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.251405 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.251478 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.251654 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.251733 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:34:42.252359 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.239256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.5.151.193:37367)
0927 18:34:42.239414 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.239452 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.240513 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.240872 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.240876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.241976 (+ 1100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.242135 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.242141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.243188 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.243189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.244557 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.244559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.245088 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.245089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.245208 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.245844 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.245853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.247781 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.250541 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.250542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.250741 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.250939 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.250940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.250992 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.251515 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.251516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.251544 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.251935 (+ 391us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.252133 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":144,"thread_start_us":91,"threads_started":1}
I20260927 18:34:42.252544 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.239530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.193:47655)
0927 18:34:42.239635 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.239644 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.240534 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.240879 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.240880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.241849 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.242135 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.242140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.243259 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.243259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.244560 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.244562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.245074 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.245075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.245218 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.245871 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.245876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.248785 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.250704 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.250705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.250913 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.251121 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.251122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.251122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.251181 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.251483 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.251484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.251536 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.251916 (+ 380us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.252178 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":189,"thread_start_us":60,"threads_started":1}
I20260927 18:34:42.252800 12805 heartbeater.cc:347] Connected to a master server at 127.5.151.253:44143
I20260927 18:34:42.252879 12805 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.253006 12805 heartbeater.cc:511] Master 127.5.151.253:44143 requested a full tablet report, sending...
I20260927 18:34:42.253473 12436 ts_manager.cc:194] Registered new tserver with Master: eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193:36311)
I20260927 18:34:42.253953 12804 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43035
I20260927 18:34:42.254002 12804 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.254068 12804 heartbeater.cc:511] Master 127.5.151.254:43035 requested a full tablet report, sending...
I20260927 18:34:42.254345 11838 ts_manager.cc:194] Re-registered known tserver with Master: eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193:36311)
I20260927 18:34:42.254905 11838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:47655
I20260927 18:34:42.254210 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.239256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.5.151.193:45443)
0927 18:34:42.239421 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.239451 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.240534 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.240879 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.240880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.241849 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.242135 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.242140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.243259 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.243259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.244560 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.244562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.245080 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.245080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.245209 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.245982 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.245987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.248132 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.249983 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.249989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.249992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.250202 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.250416 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.250417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.250417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.250557 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.250884 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.250891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.251051 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.253906 (+ 2855us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.254032 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":126,"thread_start_us":115,"threads_started":1}
I20260927 18:34:42.255945 12803 heartbeater.cc:347] Connected to a master server at 127.5.151.252:44941
I20260927 18:34:42.255995 12803 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.256052 12803 heartbeater.cc:511] Master 127.5.151.252:44941 requested a full tablet report, sending...
I20260927 18:34:42.256343 12622 ts_manager.cc:194] Registered new tserver with Master: eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193:36311)
W20260927 18:34:42.329507 12810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:42.329720 12810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:42.329756 12810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:42.330929 12810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:42.330988 12810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:42.331030 12810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:42.332458 12810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab
--principal=oryx/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:36181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=37193
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:42.332731 12810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:42.332923 12810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:42.337453 12818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:42.337515 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:42.337550 12810 server_base.cc:1034] running on GCE node
W20260927 18:34:42.337455 12815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:42.337919 12810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:42.338155 12810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:42.339305 12810 hybrid_clock.cc:648] HybridClock initialized: now 1790534082339296 us; error 23 us; skew 500 ppm
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:42.341795 12810 init.cc:377] Logged in from keytab as oryx/127.5.151.194@KRBTEST.COM (short username oryx)
I20260927 18:34:42.342130 12810 webserver.cc:460] Webserver started at http://127.5.151.194:37193/ using document root <none> and password file <none>
I20260927 18:34:42.342306 12810 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:42.342355 12810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:42.343606 12810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:42.344107 12825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.344250 12810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:42.344331 12810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "de3327d4b00a4485b9f77b16b80b8365"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "108d3ed1654327f9cfd11c57bfc78073"
server_key_iv: "2fa1e311243952eefd83441b0a0f0f4b"
server_key_version: "encryptionkey@0"
I20260927 18:34:42.344612 12810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:42.359375 12810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:42.361040 12810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:42.361169 12810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:42.361393 12810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:42.361663 12810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:42.361707 12810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.361727 12810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:42.361739 12810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.367575 12810 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:36181
I20260927 18:34:42.367645 12938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:36181 every 8 connection(s)
I20260927 18:34:42.367957 12810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:42.374060 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 12810
I20260927 18:34:42.374233 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 12168
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.253@KRBTEST.COM
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.252@KRBTEST.COM
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:42.379577 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.368643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:40837 (local address 127.5.151.252:44941)
0927 18:34:42.368691 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.368693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.368889 (+ 196us) server_negotiation.cc:407] Connection header received
0927 18:34:42.369862 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.369863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.369875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.369920 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.370685 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.371426 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372314 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.372599 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372659 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.376258 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.376260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.376260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.376267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.378055 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378651 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.378652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.378652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.378718 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378946 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.378947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.378947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.379064 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.379115 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.379410 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.379489 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:34:42.379577 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.368943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:43687 (local address 127.5.151.253:44143)
0927 18:34:42.368992 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.368994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.369041 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:34:42.369884 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.369885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.369897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.369929 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.370649 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.371413 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372431 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.372676 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372732 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.376409 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.376410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.376411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.376417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.378177 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378650 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.378651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.378652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.378733 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378955 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.378955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.378956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.379064 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.379115 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.379410 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.379489 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 18:34:42.379624 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.368583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:36957 (local address 127.5.151.254:43035)
0927 18:34:42.368663 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.368666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.368860 (+ 194us) server_negotiation.cc:407] Connection header received
0927 18:34:42.369884 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.369885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.369898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.369929 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.370682 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.371128 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372452 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.372755 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372848 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.376242 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.376243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.376244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.376250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.377928 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378516 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.378517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.378518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.378601 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378881 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.378882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.378882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.379017 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.379058 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.379406 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.379489 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:34:42.380100 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.368585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.5.151.194:40837)
0927 18:34:42.368811 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.368819 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.369769 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.370005 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.370010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.370366 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.370556 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.370560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.371512 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.371513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.372158 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.372159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.372676 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.372827 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.373469 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.373474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.376139 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.378149 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.378151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.378566 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.378795 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.378795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.378872 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.379149 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.379150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.379258 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.379901 (+ 643us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.379975 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":7,"thread_start_us":87,"threads_started":1}
I20260927 18:34:42.380103 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.368866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.5.151.194:43687)
0927 18:34:42.368993 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.369001 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.369775 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.370008 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.370011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.370375 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.370557 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.370580 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.371491 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.371492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.372332 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.372334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.372803 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.372850 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.373439 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.373449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.376321 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.378251 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.378252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.378545 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.378796 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.378797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.378884 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.379149 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.379150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.379258 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.379661 (+ 403us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.379867 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":9,"thread_start_us":54,"threads_started":1}
I20260927 18:34:42.380725 12941 heartbeater.cc:347] Connected to a master server at 127.5.151.253:44143
I20260927 18:34:42.380832 12941 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.380841 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.368586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.194:36957)
0927 18:34:42.368751 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.368790 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.369770 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.370006 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.370010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.370360 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.370557 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.370563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.371274 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.371275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.372382 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.372384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.372830 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.372832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.372902 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.373650 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.373654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.376135 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.378012 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.378020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.378432 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.378671 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.378672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.378672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.378809 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.379101 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.379107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.379258 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.379990 (+ 732us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.380102 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260927 18:34:42.381028 12941 heartbeater.cc:511] Master 127.5.151.253:44143 requested a full tablet report, sending...
I20260927 18:34:42.381150 12940 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43035
I20260927 18:34:42.381194 12940 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.381263 12940 heartbeater.cc:511] Master 127.5.151.254:43035 requested a full tablet report, sending...
I20260927 18:34:42.381414 12436 ts_manager.cc:194] Registered new tserver with Master: de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181)
I20260927 18:34:42.381458 11838 ts_manager.cc:194] Re-registered known tserver with Master: de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181)
I20260927 18:34:42.381961 11838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:36957
I20260927 18:34:42.382594 12939 heartbeater.cc:347] Connected to a master server at 127.5.151.252:44941
I20260927 18:34:42.382638 12939 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.382655 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:41229
--local_ip_for_outbound_sockets=127.5.151.195
--tserver_master_addrs=127.5.151.254:43035
--webserver_port=35597
--webserver_interface=127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab
--principal=oryx/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:34:42.382687 12939 heartbeater.cc:511] Master 127.5.151.252:44941 requested a full tablet report, sending...
I20260927 18:34:42.382978 12622 ts_manager.cc:194] Registered new tserver with Master: de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181)
W20260927 18:34:42.459514 12946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:42.459703 12946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:42.459725 12946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:42.460892 12946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:42.460932 12946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:42.460958 12946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:42.462450 12946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab
--principal=oryx/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:41229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=35597
--tserver_master_addrs=127.5.151.254:43035,127.5.151.253:44143,127.5.151.252:44941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:42.462706 12946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:42.462884 12946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:42.467659 12952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:42.467679 12951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:42.467694 12946 server_base.cc:1034] running on GCE node
W20260927 18:34:42.467677 12954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:42.468024 12946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:42.468276 12946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:42.469437 12946 hybrid_clock.cc:648] HybridClock initialized: now 1790534082469430 us; error 39 us; skew 500 ppm
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:42.471971 12946 init.cc:377] Logged in from keytab as oryx/127.5.151.195@KRBTEST.COM (short username oryx)
I20260927 18:34:42.472404 12946 webserver.cc:460] Webserver started at http://127.5.151.195:35597/ using document root <none> and password file <none>
I20260927 18:34:42.472692 12946 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:42.472772 12946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:42.474041 12946 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:42.474689 12961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.474864 12946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 18:34:42.474957 12946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "19f521890638442e9dee53f3f9156239"
format_stamp: "Formatted at 2026-09-27 18:34:41 on dist-test-slave-19nl"
server_key: "95cac4ef574dda873419d644cbe38709"
server_key_iv: "7e7b263d69d472acc50fb26b89b30a08"
server_key_version: "encryptionkey@0"
I20260927 18:34:42.475281 12946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:42.492008 12946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:42.493880 12946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:42.494042 12946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:42.494269 12946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:42.494644 12946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:42.494702 12946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.494753 12946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:42.494781 12946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:42.500681 12946 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:41229
I20260927 18:34:42.500731 13074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:41229 every 8 connection(s)
I20260927 18:34:42.501075 12946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:42.507360 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 12946
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.252@KRBTEST.COM
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
Sep 27 18:34:42 dist-test-slave-19nl krb5kdc[11793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534082, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.253@KRBTEST.COM
I20260927 18:34:42.512907 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.501772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:53859 (local address 127.5.151.254:43035)
0927 18:34:42.501821 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.501823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.502157 (+ 334us) server_negotiation.cc:407] Connection header received
0927 18:34:42.502743 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.502744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.502758 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.502790 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.503513 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.504310 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505198 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.505518 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505590 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.509439 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.509441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.509441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.509448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.511217 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.511698 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.511699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.511700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.511759 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.512053 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.512054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.512054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.512190 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.512245 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.512720 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.512798 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:34:42.513108 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.501788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.195:53859)
0927 18:34:42.501972 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.501995 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.502598 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.502873 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.502874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.503204 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.503410 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.503415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.504396 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.504397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.505079 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.505081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.505602 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.505757 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.506380 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.506390 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.509347 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.511320 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.511327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.511329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.511615 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.511845 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.511846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.511846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.511974 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.512287 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.512292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.512445 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.512552 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.512859 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":21,"thread_start_us":95,"threads_started":1}
I20260927 18:34:42.513578 13076 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43035
I20260927 18:34:42.513666 13076 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.513864 13076 heartbeater.cc:511] Master 127.5.151.254:43035 requested a full tablet report, sending...
I20260927 18:34:42.514231 11838 ts_manager.cc:194] Re-registered known tserver with Master: 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:42.514724 11838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:53859
I20260927 18:34:42.514834 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.502194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:58829 (local address 127.5.151.252:44941)
0927 18:34:42.502235 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.502237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.502245 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:42.502694 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.502696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.502710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.502761 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.503571 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.504286 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505700 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.505877 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505956 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.510031 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.510033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.510033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.510041 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.512288 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.513611 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.513612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.513613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.513673 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.514190 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.514191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.514191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.514340 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.514383 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.514638 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.514713 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:42.515149 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.501789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.5.151.195:58829)
0927 18:34:42.501950 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.501990 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.502590 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.502829 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.502832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.503416 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.503469 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.503470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.504372 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.504373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.505209 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.505210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.505957 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.506044 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.506557 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.506581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.509650 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.512395 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.512396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.512396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.513522 (+ 1126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.514034 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.514035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.514036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.514098 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.514437 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.514438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.514480 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.514904 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.515020 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":23,"thread_start_us":83,"threads_started":1}
I20260927 18:34:42.515491 13075 heartbeater.cc:347] Connected to a master server at 127.5.151.252:44941
I20260927 18:34:42.515535 13075 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.515600 13075 heartbeater.cc:511] Master 127.5.151.252:44941 requested a full tablet report, sending...
I20260927 18:34:42.515866 12622 ts_manager.cc:194] Registered new tserver with Master: 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:42.516682 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.502211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:51239 (local address 127.5.151.253:44143)
0927 18:34:42.502267 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.502268 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.502275 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:42.502823 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.502825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.502836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.502893 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.503525 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.504027 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505279 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.505582 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505642 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.509665 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.509666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.509667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.509673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.515079 (+ 5406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.515837 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.515838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.515838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.515897 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.516144 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.516145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.516145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.516266 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.516347 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.516516 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.516595 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:34:42.516695 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.502079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.5.151.195:51239)
0927 18:34:42.502191 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.502202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.502582 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.502918 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.502919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.503211 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:42.503413 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.503415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.504126 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.504126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.505193 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.505196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.505649 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.505650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.505757 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.506759 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:42.506764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:42.509575 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:42.515161 (+ 5586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.515161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.515162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.515374 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.516015 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:42.516015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:42.516016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:42.516077 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:42.516345 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:42.516346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:42.516392 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.516486 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.516595 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260927 18:34:42.517355 13077 heartbeater.cc:347] Connected to a master server at 127.5.151.253:44143
I20260927 18:34:42.517398 13077 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:42.517443 13077 heartbeater.cc:511] Master 127.5.151.253:44143 requested a full tablet report, sending...
I20260927 18:34:42.517793 12436 ts_manager.cc:194] Registered new tserver with Master: 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:42.518195 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.508864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.5.151.254:43035)
0927 18:34:42.508990 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.508991 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.508998 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:42.509018 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.509019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.509030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.509097 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.509930 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.510739 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.511878 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.512215 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.512555 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.513786 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.513787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.513788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.513794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.515745 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.516142 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.516143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.516143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.516192 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.516402 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.516403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.516404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.516520 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.517946 (+ 1426us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.517990 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.518071 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260927 18:34:42.519892 11838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:42.521504 11838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:42.523365 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.508870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.5.151.252:44941)
0927 18:34:42.510317 (+ 1447us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.510319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.510327 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:42.510348 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.510349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.510360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.510420 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.516029 (+ 5609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.517153 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.518079 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.518343 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.518400 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.520269 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.520270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.520270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.520276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.521788 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.522206 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.522207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.522208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.522273 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.522605 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.522606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.522607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.522737 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.522883 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.523175 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.523268 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1424,"thread_start_us":53,"threads_started":1}
I20260927 18:34:42.528436 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.509886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.5.151.253:44143)
0927 18:34:42.513802 (+ 3916us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.513804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.517730 (+ 3926us) server_negotiation.cc:407] Connection header received
0927 18:34:42.517748 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.517749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.517758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.517844 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.518931 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.521920 (+ 2989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.522986 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.523261 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.523670 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.524772 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.524773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.524774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.524780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.526823 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.527161 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.527162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.527163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.527221 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.527439 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.527440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.527440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.527553 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.527970 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.528006 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.528078 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3895,"thread_start_us":49,"threads_started":1}
I20260927 18:34:42.530254 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.526102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:41229 (local address 127.0.0.1:47262)
0927 18:34:42.526743 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.526751 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.526790 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.527028 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.527029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.527031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.527073 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.527075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.528232 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.528233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.529159 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.529162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.530009 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.530010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.530070 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.530074 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.530138 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.530178 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":30,"thread_start_us":62,"threads_started":1}
I20260927 18:34:42.530278 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.526284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.5.151.195:41229)
0927 18:34:42.526444 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.526449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.526850 (+ 401us) server_negotiation.cc:407] Connection header received
0927 18:34:42.526889 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.526891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.526904 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.526971 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.527149 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.528142 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.529246 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.529932 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.530003 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.530069 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.530152 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.530188 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260927 18:34:42.530763 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.525906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:36181 (local address 127.0.0.1:44526)
0927 18:34:42.526462 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.526472 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.526520 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.527172 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.527173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.527174 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.527214 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.527216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.528400 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.528400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.529429 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.529432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.530527 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.530528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.530606 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.530610 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.530658 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.530699 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":79,"threads_started":1}
I20260927 18:34:42.531011 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.526102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.5.151.194:36181)
0927 18:34:42.526642 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.526645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.526655 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:42.526699 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.526701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.526717 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.526777 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.527290 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.528320 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.529518 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.530200 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.530259 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.530326 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.530892 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.530935 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":76,"threads_started":1}
I20260927 18:34:42.530974 13009 tablet_service.cc:1467] Processing CreateTablet for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd (DEFAULT_TABLE table=test-table [id=05fad01ff5294f6783b6f700cb370713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:42.531371 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:42.531565 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.525979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36311 (local address 127.0.0.1:55414)
0927 18:34:42.526618 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.526627 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.526674 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.526914 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.526915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.526917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.526966 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.526968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.528196 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.528197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.529786 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.529788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.531340 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.531341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.531410 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.531414 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.531466 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.531503 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":893,"thread_start_us":82,"threads_started":1}
I20260927 18:34:42.531632 12873 tablet_service.cc:1467] Processing CreateTablet for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd (DEFAULT_TABLE table=test-table [id=05fad01ff5294f6783b6f700cb370713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:42.531849 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.526068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.5.151.193:36311)
0927 18:34:42.526238 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.526415 (+ 177us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.526681 (+ 266us) server_negotiation.cc:407] Connection header received
0927 18:34:42.526735 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.526738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.526756 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.526842 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.527045 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.528096 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.529876 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.531037 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.531212 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.531304 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.531730 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.531774 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260927 18:34:42.531999 12873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:42.534425 13105 tablet_bootstrap.cc:492] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Bootstrap starting.
I20260927 18:34:42.534909 12737 tablet_service.cc:1467] Processing CreateTablet for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd (DEFAULT_TABLE table=test-table [id=05fad01ff5294f6783b6f700cb370713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:42.535142 13105 tablet_bootstrap.cc:654] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:42.535163 13106 tablet_bootstrap.cc:492] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Bootstrap starting.
I20260927 18:34:42.535269 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:42.535449 13105 log.cc:824] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:42.535830 13106 tablet_bootstrap.cc:654] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:42.536093 13106 log.cc:824] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:42.536228 13105 tablet_bootstrap.cc:492] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: No bootstrap required, opened a new log
I20260927 18:34:42.536298 13105 ts_tablet_manager.cc:1397] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:42.536748 13106 tablet_bootstrap.cc:492] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: No bootstrap required, opened a new log
I20260927 18:34:42.536824 13106 ts_tablet_manager.cc:1397] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:42.538317 13106 raft_consensus.cc:348] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.538451 13106 raft_consensus.cc:374] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:42.538491 13106 raft_consensus.cc:729] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3327d4b00a4485b9f77b16b80b8365, State: Initialized, Role: FOLLOWER
I20260927 18:34:42.538635 13106 consensus_queue.cc:260] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.538867 13106 ts_tablet_manager.cc:1428] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:42.538856 13109 tablet_bootstrap.cc:492] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Bootstrap starting.
I20260927 18:34:42.538815 13105 raft_consensus.cc:348] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.538957 13105 raft_consensus.cc:374] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:42.539000 13105 raft_consensus.cc:729] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f521890638442e9dee53f3f9156239, State: Initialized, Role: FOLLOWER
I20260927 18:34:42.539122 13105 consensus_queue.cc:260] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.539182 12940 heartbeater.cc:503] Master 127.5.151.254:43035 was elected leader, sending a full tablet report...
I20260927 18:34:42.539356 13105 ts_tablet_manager.cc:1428] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:34:42.539467 13109 tablet_bootstrap.cc:654] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:42.539741 13109 log.cc:824] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:42.539772 13076 heartbeater.cc:503] Master 127.5.151.254:43035 was elected leader, sending a full tablet report...
I20260927 18:34:42.540670 13109 tablet_bootstrap.cc:492] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: No bootstrap required, opened a new log
I20260927 18:34:42.540728 13109 ts_tablet_manager.cc:1397] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:42.542497 13109 raft_consensus.cc:348] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.542681 13109 raft_consensus.cc:374] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:42.542706 13109 raft_consensus.cc:729] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4d3886c1d544d984c4ba30ddb45751, State: Initialized, Role: FOLLOWER
I20260927 18:34:42.542781 13109 consensus_queue.cc:260] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.542992 13109 ts_tablet_manager.cc:1428] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:34:42.543025 12804 heartbeater.cc:503] Master 127.5.151.254:43035 was elected leader, sending a full tablet report...
I20260927 18:34:42.609462 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.606948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.5.151.252:44941)
0927 18:34:42.607017 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.607019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.607028 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:42.607061 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.607062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.607088 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.607160 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.607336 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.607802 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.608747 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.609153 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.609235 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.609317 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.609346 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.609388 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:34:42.609462 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.606871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.0.0.1:37298)
0927 18:34:42.606939 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.606951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.607002 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.607181 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.607182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.607184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.607239 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.607241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.607884 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.607886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.608652 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.608654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.609236 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.609237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.609295 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.609299 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.609341 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.609390 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:34:42.609862 12632 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 18:34:42.610769 13114 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a85abd132ad4174a3206e8898af3222. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.610877 13114 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: This master's current role is: LEARNER
W20260927 18:34:42.619009 12942 tablet.cc:2369] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:42.739455 12806 tablet.cc:2369] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:34:42.751860 13078 tablet.cc:2369] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:42.766523 13113 raft_consensus.cc:482] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:42.766650 13113 raft_consensus.cc:504] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.766896 13111 raft_consensus.cc:482] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:42.767063 13113 leader_election.cc:288] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181), 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:42.767031 13111 raft_consensus.cc:504] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.767656 13111 leader_election.cc:288] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181), eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193:36311)
I20260927 18:34:42.770886 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.767266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:35747 (local address 127.5.151.194:36181)
0927 18:34:42.767310 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.767313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.767438 (+ 125us) server_negotiation.cc:407] Connection header received
0927 18:34:42.767499 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.767500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.767510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.767582 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.767775 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.768514 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.769986 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.770503 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770570 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.770578 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.770784 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.770823 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:42.770884 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.767402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:37935 (local address 127.5.151.195:41229)
0927 18:34:42.767440 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.767442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.767586 (+ 144us) server_negotiation.cc:407] Connection header received
0927 18:34:42.767680 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.767681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.767689 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.767718 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.767882 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.768550 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.769987 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.770504 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770570 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.770578 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.770784 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.770819 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:42.771117 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.767228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:41229 (local address 127.5.151.193:37935)
0927 18:34:42.767494 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.767502 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.767611 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.767777 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.767778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.767779 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.767815 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.767816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.768631 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.768631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.769884 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.769886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.770640 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.770706 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.770708 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.770990 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.771019 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":85,"thread_start_us":53,"threads_started":1}
I20260927 18:34:42.771207 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.768051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36311 (local address 127.5.151.195:46043)
0927 18:34:42.768126 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.768135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.768166 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.768386 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.768386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.768387 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.768437 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.768438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.769215 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.769216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.770225 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.770228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.770953 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.771044 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.771047 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.771090 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.771127 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:34:42.771219 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.767202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:36181 (local address 127.5.151.193:35747)
0927 18:34:42.767358 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.767369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.767438 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.767627 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.767628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.767630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.767688 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.767690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.768608 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.768609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.769884 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.769886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.770638 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.770703 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.770706 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.770959 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.771006 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":38,"thread_start_us":73,"threads_started":1}
I20260927 18:34:42.771569 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.768230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:46043 (local address 127.5.151.193:36311)
0927 18:34:42.768269 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.768270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.768278 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:42.768298 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.768299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.768306 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.768352 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.768500 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.769143 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770321 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.770849 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770914 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.770922 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.771473 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.771512 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":23}
I20260927 18:34:42.771549 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f521890638442e9dee53f3f9156239" is_pre_election: true
I20260927 18:34:42.771659 13029 raft_consensus.cc:2454] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4d3886c1d544d984c4ba30ddb45751 in term 0.
I20260927 18:34:42.771697 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.767833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:49877 (local address 127.5.151.194:36181)
0927 18:34:42.767973 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.767975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.767983 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:42.768002 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.768003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.768012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.768227 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:42.768258 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.769018 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.770335 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.770901 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.771083 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.771091 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.771595 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.771631 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260927 18:34:42.771565 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.767743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:36181 (local address 127.5.151.195:49877)
0927 18:34:42.767788 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:42.767797 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:42.767835 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:42.768100 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:42.768101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:42.768102 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:42.768165 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.768167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.769103 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.769104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.770226 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:42.770228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.771269 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.771270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:42.771375 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.771379 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:42.771428 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:34:42.771474 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260927 18:34:42.771768 12757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "19f521890638442e9dee53f3f9156239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4d3886c1d544d984c4ba30ddb45751" is_pre_election: true
I20260927 18:34:42.771828 12690 leader_election.cc:302] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f521890638442e9dee53f3f9156239, eb4d3886c1d544d984c4ba30ddb45751; no voters:
I20260927 18:34:42.771808 12893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3327d4b00a4485b9f77b16b80b8365" is_pre_election: true
I20260927 18:34:42.771870 12757 raft_consensus.cc:2454] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f521890638442e9dee53f3f9156239 in term 0.
I20260927 18:34:42.771952 12893 raft_consensus.cc:2454] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4d3886c1d544d984c4ba30ddb45751 in term 0.
I20260927 18:34:42.772018 12963 leader_election.cc:302] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f521890638442e9dee53f3f9156239, eb4d3886c1d544d984c4ba30ddb45751; no voters:
I20260927 18:34:42.772145 13111 raft_consensus.cc:2781] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:42.772215 13111 raft_consensus.cc:482] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:42.772181 12892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "19f521890638442e9dee53f3f9156239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3327d4b00a4485b9f77b16b80b8365" is_pre_election: true
I20260927 18:34:42.772238 13111 raft_consensus.cc:3037] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:42.772254 12892 raft_consensus.cc:2454] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f521890638442e9dee53f3f9156239 in term 0.
I20260927 18:34:42.772239 13113 raft_consensus.cc:2781] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:42.772296 13113 raft_consensus.cc:482] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:42.772347 13113 raft_consensus.cc:3037] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:42.772998 13111 raft_consensus.cc:504] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.773041 13113 raft_consensus.cc:504] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.773118 13111 leader_election.cc:288] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [CANDIDATE]: Term 1 election: Requested vote from peers de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181), eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193:36311)
I20260927 18:34:42.773156 13113 leader_election.cc:288] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 election: Requested vote from peers de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181), 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:42.773232 12892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "19f521890638442e9dee53f3f9156239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3327d4b00a4485b9f77b16b80b8365"
I20260927 18:34:42.773303 12892 raft_consensus.cc:3037] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:42.773372 12757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "19f521890638442e9dee53f3f9156239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4d3886c1d544d984c4ba30ddb45751"
I20260927 18:34:42.773445 12757 raft_consensus.cc:2379] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 19f521890638442e9dee53f3f9156239 in current term 1: Already voted for candidate eb4d3886c1d544d984c4ba30ddb45751 in this term.
I20260927 18:34:42.773416 12893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3327d4b00a4485b9f77b16b80b8365"
I20260927 18:34:42.773654 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d551ba1cd4f1f8e3849f1ee18a9fd" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f521890638442e9dee53f3f9156239"
I20260927 18:34:42.773758 13029 raft_consensus.cc:2379] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb4d3886c1d544d984c4ba30ddb45751 in current term 1: Already voted for candidate 19f521890638442e9dee53f3f9156239 in this term.
I20260927 18:34:42.773977 12690 leader_election.cc:302] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb4d3886c1d544d984c4ba30ddb45751; no voters: 19f521890638442e9dee53f3f9156239, de3327d4b00a4485b9f77b16b80b8365
I20260927 18:34:42.774209 13113 raft_consensus.cc:2726] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 18:34:42.774220 12892 raft_consensus.cc:2454] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f521890638442e9dee53f3f9156239 in term 1.
I20260927 18:34:42.774387 12965 leader_election.cc:302] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19f521890638442e9dee53f3f9156239, de3327d4b00a4485b9f77b16b80b8365; no voters: eb4d3886c1d544d984c4ba30ddb45751
I20260927 18:34:42.774503 13111 raft_consensus.cc:2781] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:42.774665 13111 raft_consensus.cc:686] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 1 LEADER]: Becoming Leader. State: Replica: 19f521890638442e9dee53f3f9156239, State: Running, Role: LEADER
I20260927 18:34:42.774780 13111 consensus_queue.cc:237] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } }
I20260927 18:34:42.775756 11838 catalog_manager.cc:5495] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f521890638442e9dee53f3f9156239 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "19f521890638442e9dee53f3f9156239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } health_report { overall_health: HEALTHY } } }
I20260927 18:34:42.776342 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: NON_VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: true } } }
I20260927 18:34:42.776430 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:42.777012 13124 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: attempting to promote NON_VOTER b2c6219c34b74bcd974ff88591ab1ca3 to VOTER
I20260927 18:34:42.777173 13124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } }
I20260927 18:34:42.777628 12632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 LEARNER]: Refusing update from remote peer 9a85abd132ad4174a3206e8898af3222: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 18:34:42.777688 12446 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Refusing update from remote peer 9a85abd132ad4174a3206e8898af3222: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 18:34:42.777864 13093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260927 18:34:42.777948 13098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260927 18:34:42.778599 13098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.778743 12446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.778884 12632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.779500 13098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.779584 13098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:42.779634 13098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.779687 13098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:42.779739 13098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.779714 13114 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.779783 13098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:42.779794 13114 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:34:42.779760 13126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a85abd132ad4174a3206e8898af3222" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:42.779969 13126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:34:42.810864 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.808626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.5.151.195:41229)
0927 18:34:42.808688 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.808690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.808699 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:42.808741 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.808742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.808762 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.808826 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:42.808956 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.809419 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.810100 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.810298 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.810384 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.810542 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:42.810657 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:42.810748 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.810767 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.810784 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:34:42.812551 12892 raft_consensus.cc:1261] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Refusing update from remote peer 19f521890638442e9dee53f3f9156239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:42.812551 12757 raft_consensus.cc:1261] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Refusing update from remote peer 19f521890638442e9dee53f3f9156239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:42.812827 13122 consensus_queue.cc:1035] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:42.812954 13111 consensus_queue.cc:1035] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:42.816231 13128 mvcc.cc:205] Tried to move back new op lower bound from 7334027603197337600 to 7334027603045928960. Current Snapshot: MvccSnapshot[applied={T|T < 7334027603197337600}]
I20260927 18:34:42.818581 11838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:42.818674 11838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:42.819892 11838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a85abd132ad4174a3206e8898af3222: Sending DeleteTablet for 3 replicas of tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd
I20260927 18:34:42.820204 13009 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:42 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47262
I20260927 18:34:42.820246 12873 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:42 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:44526
I20260927 18:34:42.820600 12737 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6d551ba1cd4f1f8e3849f1ee18a9fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:42 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:55414
I20260927 18:34:42.820812 13140 tablet_replica.cc:330] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: stopping tablet replica
I20260927 18:34:42.820911 13140 raft_consensus.cc:2229] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:42.821010 13140 raft_consensus.cc:2258] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:42.821255 13138 tablet_replica.cc:330] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: stopping tablet replica
I20260927 18:34:42.821288 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 11806
I20260927 18:34:42.821954 13138 raft_consensus.cc:2229] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:42.822063 13138 raft_consensus.cc:2258] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:42.822072 13140 ts_tablet_manager.cc:1905] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:42.822630 13138 ts_tablet_manager.cc:1905] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:42.823807 13140 ts_tablet_manager.cc:1918] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:42.823871 13140 log.cc:1196] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/ee6d551ba1cd4f1f8e3849f1ee18a9fd
I20260927 18:34:42.824116 13140 ts_tablet_manager.cc:1939] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P eb4d3886c1d544d984c4ba30ddb45751: Deleting consensus metadata
I20260927 18:34:42.824314 13138 ts_tablet_manager.cc:1918] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:42.824368 13138 log.cc:1196] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/ee6d551ba1cd4f1f8e3849f1ee18a9fd
I20260927 18:34:42.824601 13138 ts_tablet_manager.cc:1939] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P 19f521890638442e9dee53f3f9156239: Deleting consensus metadata
W20260927 18:34:42.827872 12965 connection.cc:535] server connection from 127.0.0.1:47262 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 18:34:42.827986 12691 connection.cc:535] server connection from 127.0.0.1:55414 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:34:42.830670 13139 tablet_replica.cc:330] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: stopping tablet replica
I20260927 18:34:42.830760 13139 raft_consensus.cc:2229] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:42.830845 13139 raft_consensus.cc:2258] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:42.831744 13139 ts_tablet_manager.cc:1905] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:42.833020 13139 ts_tablet_manager.cc:1918] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:42.833098 13139 log.cc:1196] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/ee6d551ba1cd4f1f8e3849f1ee18a9fd
I20260927 18:34:42.833367 13139 ts_tablet_manager.cc:1939] T ee6d551ba1cd4f1f8e3849f1ee18a9fd P de3327d4b00a4485b9f77b16b80b8365: Deleting consensus metadata
W20260927 18:34:42.833763 12827 connection.cc:441] server connection from 127.0.0.1:44526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44526 (request call id 1) could send its response
I20260927 18:34:42.837105 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.829564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.5.151.253:44143)
0927 18:34:42.829614 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.829616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.829754 (+ 138us) server_negotiation.cc:407] Connection header received
0927 18:34:42.829848 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.829849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.829861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.829890 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.830613 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.831361 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.832441 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.832736 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.832776 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.834325 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.834326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.834326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.834333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.835903 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.836265 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.836266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.836266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.836317 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.836523 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.836523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.836524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.836639 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.836708 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.836909 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.836994 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:34:42.837738 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:42.829447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.5.151.252:44941)
0927 18:34:42.829521 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:42.829523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:42.829677 (+ 154us) server_negotiation.cc:407] Connection header received
0927 18:34:42.829756 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:42.829757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:42.829767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:42.829830 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:42.830403 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.831092 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.832916 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:42.833626 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:42.833704 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:42.835319 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:42.835320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:42.835320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:42.835326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:42.836607 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.836995 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.836996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.836996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.837047 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:42.837253 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:42.837254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:42.837255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:42.837383 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:42.837466 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:42.837622 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 18:34:42.837683 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:34:43.817003 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:43.816719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.193:45135)
0927 18:34:43.816860 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:43.816946 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
W20260927 18:34:43.817256 12804 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:43035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
I20260927 18:34:43.817716 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:43.817477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.194:37217)
0927 18:34:43.817577 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:43.817671 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
I20260927 18:34:43.817979 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:43.817767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.195:41087)
0927 18:34:43.817858 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:43.817919 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
W20260927 18:34:43.818034 12940 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:43035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
W20260927 18:34:43.818174 13076 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:43035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
I20260927 18:34:44.320367 13151 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a85abd132ad4174a3206e8898af3222)
I20260927 18:34:44.320510 13151 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } }
I20260927 18:34:44.320824 13151 leader_election.cc:288] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a85abd132ad4174a3206e8898af3222 (127.5.151.254:43035), 0ff43d1f6acc40ef8b9c8469c74c641b (127.5.151.253:44143)
I20260927 18:34:44.320858 13152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a85abd132ad4174a3206e8898af3222)
I20260927 18:34:44.320991 13152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } }
I20260927 18:34:44.321341 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.320954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38132)
0927 18:34:44.321229 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.321299 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":60,"threads_started":1}
W20260927 18:34:44.321522 12607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
I20260927 18:34:44.321592 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.321422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38142)
0927 18:34:44.321492 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.321550 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58,"thread_start_us":38,"threads_started":1}
W20260927 18:34:44.321692 12421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
I20260927 18:34:44.321848 13152 leader_election.cc:288] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a85abd132ad4174a3206e8898af3222 (127.5.151.254:43035), b2c6219c34b74bcd974ff88591ab1ca3 (127.5.151.252:44941)
I20260927 18:34:44.322117 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.322012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38146)
0927 18:34:44.322064 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.322086 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 18:34:44.322216 12607 leader_election.cc:334] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a85abd132ad4174a3206e8898af3222 (127.5.151.254:43035): Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
I20260927 18:34:44.322341 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.322038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38148)
0927 18:34:44.322291 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.322319 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
W20260927 18:34:44.322448 12421 leader_election.cc:334] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a85abd132ad4174a3206e8898af3222 (127.5.151.254:43035): Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
I20260927 18:34:44.324229 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.321130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:44143 (local address 127.0.0.1:39096)
0927 18:34:44.321232 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.321242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:44.321315 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:44.321500 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:44.321501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:44.321503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:44.321558 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.321560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.322525 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.322526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.323492 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.323494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.324060 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.324061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.324111 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.324114 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:44.324152 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 18:34:44.324186 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":46,"threads_started":1}
I20260927 18:34:44.324268 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.321226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.5.151.253:44143)
0927 18:34:44.321336 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:44.321338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:44.321347 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:44.321374 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:44.321375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:44.321385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:44.321447 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:44.321638 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.322389 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.323615 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.323964 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.324057 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.324068 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:44.324172 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 18:34:44.324207 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
I20260927 18:34:44.324590 12446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" is_pre_election: true
I20260927 18:34:44.324719 12446 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2c6219c34b74bcd974ff88591ab1ca3 in term 1.
I20260927 18:34:44.324882 12607 leader_election.cc:302] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ff43d1f6acc40ef8b9c8469c74c641b, b2c6219c34b74bcd974ff88591ab1ca3; no voters: 9a85abd132ad4174a3206e8898af3222
I20260927 18:34:44.324916 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.321591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:44941 (local address 127.0.0.1:37320)
0927 18:34:44.321840 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.321851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:44.321956 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:44.322196 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:44.322197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:44.322199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:44.322244 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.322246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.323136 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.323137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.323934 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.323936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.324657 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.324658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.324724 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.324728 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:44.324786 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:34:44.324826 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":46,"threads_started":1}
I20260927 18:34:44.324898 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.321767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.5.151.252:44941)
0927 18:34:44.322073 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:44.322075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:44.322080 (+ 5us) server_negotiation.cc:407] Connection header received
0927 18:34:44.322098 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:44.322099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:44.322107 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:44.322165 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:44.322318 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.323061 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.324013 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.324569 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.324653 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.324663 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:44.324807 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:34:44.324842 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":62,"threads_started":1}
I20260927 18:34:44.325055 13151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 18:34:44.325093 13151 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a85abd132ad4174a3206e8898af3222)
I20260927 18:34:44.325136 13151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:34:44.325131 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" is_pre_election: true
I20260927 18:34:44.325881 13151 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } }
I20260927 18:34:44.326009 12632 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ff43d1f6acc40ef8b9c8469c74c641b in current term 2: Already voted for candidate b2c6219c34b74bcd974ff88591ab1ca3 in this term.
I20260927 18:34:44.326014 13151 leader_election.cc:288] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [CANDIDATE]: Term 2 election: Requested vote from peers 9a85abd132ad4174a3206e8898af3222 (127.5.151.254:43035), 0ff43d1f6acc40ef8b9c8469c74c641b (127.5.151.253:44143)
I20260927 18:34:44.326234 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.326110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38162)
0927 18:34:44.326189 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.326213 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 18:34:44.326232 12422 leader_election.cc:302] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ff43d1f6acc40ef8b9c8469c74c641b; no voters: 9a85abd132ad4174a3206e8898af3222, b2c6219c34b74bcd974ff88591ab1ca3
I20260927 18:34:44.326236 12446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b"
I20260927 18:34:44.326339 12446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:34:44.326987 12446 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2c6219c34b74bcd974ff88591ab1ca3 in term 2.
I20260927 18:34:44.327073 13152 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:34:44.327143 12607 leader_election.cc:302] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff43d1f6acc40ef8b9c8469c74c641b, b2c6219c34b74bcd974ff88591ab1ca3; no voters:
I20260927 18:34:44.327234 13151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:34:44.327381 13151 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [term 2 LEADER]: Becoming Leader. State: Replica: b2c6219c34b74bcd974ff88591ab1ca3, State: Running, Role: LEADER
I20260927 18:34:44.327446 13151 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } }
I20260927 18:34:44.327490 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.327371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38170)
0927 18:34:44.327433 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.327461 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 18:34:44.327612 12607 leader_election.cc:334] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a85abd132ad4174a3206e8898af3222 (127.5.151.254:43035): Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
I20260927 18:34:44.327718 13161 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c6219c34b74bcd974ff88591ab1ca3. Latest consensus state: current_term: 2 leader_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:44.327806 13161 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:44.328011 13163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:44.328392 13163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:44.328449 13163 catalog_manager.cc:1259] Loaded cluster ID: 3e1305580d214a729e724d87943fe8a1
I20260927 18:34:44.328496 13163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:44.328646 13163 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:44.328747 13163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Loaded TSK: 0
I20260927 18:34:44.328887 13163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:34:44.371945 12622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:44.373791 12622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:44.374814 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.374707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38188)
0927 18:34:44.374753 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.374783 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 18:34:44.374927 12446 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [term 2 FOLLOWER]: Refusing update from remote peer b2c6219c34b74bcd974ff88591ab1ca3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 18:34:44.375141 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.375059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.0.0.1:38194)
0927 18:34:44.375092 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.375121 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 18:34:44.375165 13151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 18:34:44.375264 12607 consensus_peers.cc:487] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 -> Peer 9a85abd132ad4174a3206e8898af3222 (127.5.151.254:43035): Couldn't send request to peer 9a85abd132ad4174a3206e8898af3222. Status: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:34:44.375726 13152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c6219c34b74bcd974ff88591ab1ca3. Latest consensus state: current_term: 2 leader_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:44.375803 13152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:34:44.376248 13161 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:44.376345 13161 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:44.376439 13152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9a85abd132ad4174a3206e8898af3222" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43035 } } peers { permanent_uuid: "0ff43d1f6acc40ef8b9c8469c74c641b" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 44143 } attrs { promote: false } } peers { permanent_uuid: "b2c6219c34b74bcd974ff88591ab1ca3" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 44941 } attrs { promote: false } } }
I20260927 18:34:44.376526 13152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff43d1f6acc40ef8b9c8469c74c641b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:34:44.376598 13165 mvcc.cc:205] Tried to move back new op lower bound from 7334027609597329408 to 7334027609405566976. Current Snapshot: MvccSnapshot[applied={T|T < 7334027609597329408 or (T in {7334027609597329408})}]
I20260927 18:34:44.381443 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.378327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.5.151.194:36181)
0927 18:34:44.378969 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:44.378971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:44.378978 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:44.379000 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:44.379001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:44.379009 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:44.379083 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:44.379220 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.379866 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.380745 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.381149 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.381227 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.381238 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:44.381347 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 18:34:44.381380 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":75,"threads_started":1}
I20260927 18:34:44.381438 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.378233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:36181 (local address 127.0.0.1:44532)
0927 18:34:44.378330 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.378341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:44.378387 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:44.379096 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:44.379097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:44.379099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:44.379148 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.379149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.379955 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.379956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.380660 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.380662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.381226 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.381226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.381287 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.381290 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:44.381339 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:34:44.381377 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 18:34:44.381642 12873 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa752eedaf546e88fb1f877bd994232 (DEFAULT_TABLE table=test-table [id=24444dffb81044cc9e73a910f05df7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:44.381742 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.378286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:41229 (local address 127.0.0.1:47274)
0927 18:34:44.378482 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.378492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:44.378540 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:44.378774 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:44.378775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:44.378776 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:44.378825 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.378826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.379717 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.379717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.380733 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.380735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.381423 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.381424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.381597 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.381600 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:44.381626 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 18:34:44.381661 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":19}
I20260927 18:34:44.381765 12873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa752eedaf546e88fb1f877bd994232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:44.381798 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.378491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.5.151.193:36311)
0927 18:34:44.378603 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:44.378604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:44.378704 (+ 100us) server_negotiation.cc:407] Connection header received
0927 18:34:44.378745 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:44.378746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:44.378757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:44.378817 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:44.378957 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.379539 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.380887 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.381393 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.381443 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.381453 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:44.381687 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 18:34:44.381727 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":47,"threads_started":1}
I20260927 18:34:44.381942 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.378338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36311 (local address 127.0.0.1:55426)
0927 18:34:44.378644 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.378653 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:44.378696 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:44.378857 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:44.378858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:44.378860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:44.378890 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.378891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.379613 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.379613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.380796 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:44.380797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.381478 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.381479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:44.381534 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.381538 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:44.381584 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 18:34:44.381629 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":9,"thread_start_us":60,"threads_started":1}
I20260927 18:34:44.381965 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.378391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.5.151.195:41229)
0927 18:34:44.378503 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:44.378505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:44.378526 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:34:44.378620 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:44.378621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:44.378631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:44.378726 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:44.378880 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.379644 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.380818 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.381327 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.381837 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.381849 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:44.381867 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 18:34:44.381902 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260927 18:34:44.382112 13009 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa752eedaf546e88fb1f877bd994232 (DEFAULT_TABLE table=test-table [id=24444dffb81044cc9e73a910f05df7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:44.382210 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa752eedaf546e88fb1f877bd994232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:44.382201 12737 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa752eedaf546e88fb1f877bd994232 (DEFAULT_TABLE table=test-table [id=24444dffb81044cc9e73a910f05df7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:44.382287 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa752eedaf546e88fb1f877bd994232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:44.383201 13174 tablet_bootstrap.cc:492] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: Bootstrap starting.
I20260927 18:34:44.383512 13174 tablet_bootstrap.cc:654] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:44.383669 13176 tablet_bootstrap.cc:492] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: Bootstrap starting.
I20260927 18:34:44.384084 13176 tablet_bootstrap.cc:654] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:44.384150 13174 tablet_bootstrap.cc:492] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: No bootstrap required, opened a new log
I20260927 18:34:44.384191 13174 ts_tablet_manager.cc:1397] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:44.384346 13175 tablet_bootstrap.cc:492] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751: Bootstrap starting.
I20260927 18:34:44.384347 13174 raft_consensus.cc:348] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.384410 13174 raft_consensus.cc:374] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:44.384430 13174 raft_consensus.cc:729] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3327d4b00a4485b9f77b16b80b8365, State: Initialized, Role: FOLLOWER
I20260927 18:34:44.384481 13174 consensus_queue.cc:260] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.384629 13174 ts_tablet_manager.cc:1428] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:44.384855 13176 tablet_bootstrap.cc:492] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: No bootstrap required, opened a new log
I20260927 18:34:44.384910 13176 ts_tablet_manager.cc:1397] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:44.385003 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.384773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.194:51703)
0927 18:34:44.384922 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.384964 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260927 18:34:44.385027 13176 raft_consensus.cc:348] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.385107 13176 raft_consensus.cc:374] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:44.385136 13176 raft_consensus.cc:729] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f521890638442e9dee53f3f9156239, State: Initialized, Role: FOLLOWER
I20260927 18:34:44.385197 13176 consensus_queue.cc:260] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.385331 13176 ts_tablet_manager.cc:1428] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:44.385465 13175 tablet_bootstrap.cc:654] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:44.385790 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.385593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.195:35939)
0927 18:34:44.385720 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.385754 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260927 18:34:44.386020 12939 heartbeater.cc:503] Master 127.5.151.252:44941 was elected leader, sending a full tablet report...
I20260927 18:34:44.386356 13075 heartbeater.cc:503] Master 127.5.151.252:44941 was elected leader, sending a full tablet report...
I20260927 18:34:44.386349 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.386256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.194:57295)
0927 18:34:44.386299 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.386324 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 18:34:44.386771 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.386678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.195:48165)
0927 18:34:44.386723 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.386748 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:34:44.387125 13175 tablet_bootstrap.cc:492] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751: No bootstrap required, opened a new log
I20260927 18:34:44.387163 13175 ts_tablet_manager.cc:1397] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:34:44.387322 13175 raft_consensus.cc:348] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.387372 13175 raft_consensus.cc:374] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:44.387408 13175 raft_consensus.cc:729] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4d3886c1d544d984c4ba30ddb45751, State: Initialized, Role: FOLLOWER
I20260927 18:34:44.387463 13175 consensus_queue.cc:260] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.387611 13175 ts_tablet_manager.cc:1428] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:44.387981 12803 heartbeater.cc:503] Master 127.5.151.252:44941 was elected leader, sending a full tablet report...
I20260927 18:34:44.387984 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.387785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.193:60337)
0927 18:34:44.387889 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.387943 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":68,"threads_started":1}
I20260927 18:34:44.388345 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.388243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.193:52239)
0927 18:34:44.388296 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.388323 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:34:44.504061 13184 raft_consensus.cc:482] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:44.504155 13184 raft_consensus.cc:504] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.504328 13184 leader_election.cc:288] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181), 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:44.504566 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa752eedaf546e88fb1f877bd994232" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f521890638442e9dee53f3f9156239" is_pre_election: true
I20260927 18:34:44.504688 13029 raft_consensus.cc:2454] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4d3886c1d544d984c4ba30ddb45751 in term 0.
I20260927 18:34:44.504663 12892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa752eedaf546e88fb1f877bd994232" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3327d4b00a4485b9f77b16b80b8365" is_pre_election: true
I20260927 18:34:44.504746 12892 raft_consensus.cc:2454] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4d3886c1d544d984c4ba30ddb45751 in term 0.
I20260927 18:34:44.504922 12690 leader_election.cc:302] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f521890638442e9dee53f3f9156239, eb4d3886c1d544d984c4ba30ddb45751; no voters:
I20260927 18:34:44.505043 13184 raft_consensus.cc:2781] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:44.505082 13184 raft_consensus.cc:482] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:44.505110 13184 raft_consensus.cc:3037] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:44.505785 13184 raft_consensus.cc:504] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.505942 13184 leader_election.cc:288] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 election: Requested vote from peers de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181), 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229)
I20260927 18:34:44.506116 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa752eedaf546e88fb1f877bd994232" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f521890638442e9dee53f3f9156239"
I20260927 18:34:44.506106 12892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa752eedaf546e88fb1f877bd994232" candidate_uuid: "eb4d3886c1d544d984c4ba30ddb45751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3327d4b00a4485b9f77b16b80b8365"
I20260927 18:34:44.506189 13029 raft_consensus.cc:3037] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:44.506215 12892 raft_consensus.cc:3037] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:44.507001 12892 raft_consensus.cc:2454] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4d3886c1d544d984c4ba30ddb45751 in term 1.
I20260927 18:34:44.507031 13029 raft_consensus.cc:2454] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4d3886c1d544d984c4ba30ddb45751 in term 1.
I20260927 18:34:44.507164 12693 leader_election.cc:302] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de3327d4b00a4485b9f77b16b80b8365, eb4d3886c1d544d984c4ba30ddb45751; no voters:
I20260927 18:34:44.507269 13184 raft_consensus.cc:2781] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:44.507467 13184 raft_consensus.cc:686] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [term 1 LEADER]: Becoming Leader. State: Replica: eb4d3886c1d544d984c4ba30ddb45751, State: Running, Role: LEADER
I20260927 18:34:44.507550 13184 consensus_queue.cc:237] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } }
I20260927 18:34:44.507839 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.507732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.193:54517)
0927 18:34:44.507767 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.507807 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 18:34:44.508284 12622 catalog_manager.cc:5495] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "eb4d3886c1d544d984c4ba30ddb45751" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb4d3886c1d544d984c4ba30ddb45751" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36311 } health_report { overall_health: HEALTHY } } }
I20260927 18:34:44.525285 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.522867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.5.151.193:36311)
0927 18:34:44.522921 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:44.522923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:44.522933 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:44.523008 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:44.523009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:44.523029 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:44.523116 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:44.523266 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.523769 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.524508 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:44.524708 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:44.524787 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:44.524946 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:44.525054 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:44.525127 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:44.525178 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 18:34:44.525201 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:34:44.526306 12892 raft_consensus.cc:1261] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Refusing update from remote peer eb4d3886c1d544d984c4ba30ddb45751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:44.526422 13029 raft_consensus.cc:1261] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Refusing update from remote peer eb4d3886c1d544d984c4ba30ddb45751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:44.526693 13184 consensus_queue.cc:1035] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3327d4b00a4485b9f77b16b80b8365" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:44.526849 13186 consensus_queue.cc:1035] T 7aa752eedaf546e88fb1f877bd994232 P eb4d3886c1d544d984c4ba30ddb45751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f521890638442e9dee53f3f9156239" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:44.527786 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.527515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.194:59575)
0927 18:34:44.527710 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.527750 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260927 18:34:44.527930 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.527647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.195:49057)
0927 18:34:44.527870 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.527897 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260927 18:34:44.529115 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.528957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.193:46195)
0927 18:34:44.529060 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.529089 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 18:34:44.529325 13190 mvcc.cc:205] Tried to move back new op lower bound from 7334027610217750528 to 7334027610143444992. Current Snapshot: MvccSnapshot[applied={T|T < 7334027610217750528 or (T in {7334027610217750528})}]
I20260927 18:34:44.529462 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.529338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.195:57139)
0927 18:34:44.529408 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.529432 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:34:44.530210 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.530121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43035 (local address 127.5.151.194:50727)
0927 18:34:44.530154 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.530179 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 18:34:44.531708 12621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:44.531795 12621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:44.533066 12621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2c6219c34b74bcd974ff88591ab1ca3: Sending DeleteTablet for 3 replicas of tablet 7aa752eedaf546e88fb1f877bd994232
I20260927 18:34:44.533390 12737 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa752eedaf546e88fb1f877bd994232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:44 UTC) from {username='oryx', principal='oryx/127.5.151.252@KRBTEST.COM'} at 127.0.0.1:55426
I20260927 18:34:44.533560 13009 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa752eedaf546e88fb1f877bd994232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:44 UTC) from {username='oryx', principal='oryx/127.5.151.252@KRBTEST.COM'} at 127.0.0.1:47274
I20260927 18:34:44.533747 13200 tablet_replica.cc:330] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: stopping tablet replica
I20260927 18:34:44.533806 13200 raft_consensus.cc:2229] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:44.533849 13200 raft_consensus.cc:2258] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:44.533953 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 12674
I20260927 18:34:44.534701 13200 ts_tablet_manager.cc:1905] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:44.536656 13200 ts_tablet_manager.cc:1918] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:44.536733 13200 log.cc:1196] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/7aa752eedaf546e88fb1f877bd994232
I20260927 18:34:44.536985 13200 ts_tablet_manager.cc:1939] T 7aa752eedaf546e88fb1f877bd994232 P 19f521890638442e9dee53f3f9156239: Deleting consensus metadata
I20260927 18:34:44.537310 12606 catalog_manager.cc:4841] TS 19f521890638442e9dee53f3f9156239 (127.5.151.195:41229): tablet 7aa752eedaf546e88fb1f877bd994232 (table test-table [id=24444dffb81044cc9e73a910f05df7bb]) successfully deleted
I20260927 18:34:44.537886 12873 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa752eedaf546e88fb1f877bd994232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:44 UTC) from {username='oryx', principal='oryx/127.5.151.252@KRBTEST.COM'} at 127.0.0.1:44532
I20260927 18:34:44.538061 13201 tablet_replica.cc:330] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: stopping tablet replica
I20260927 18:34:44.538138 13201 raft_consensus.cc:2229] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:44.538188 13201 raft_consensus.cc:2258] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:44.538460 13201 ts_tablet_manager.cc:1905] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:44.539870 13201 ts_tablet_manager.cc:1918] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:44.539934 13201 log.cc:1196] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/7aa752eedaf546e88fb1f877bd994232
I20260927 18:34:44.540145 13201 ts_tablet_manager.cc:1939] T 7aa752eedaf546e88fb1f877bd994232 P de3327d4b00a4485b9f77b16b80b8365: Deleting consensus metadata
I20260927 18:34:44.540486 12609 catalog_manager.cc:4841] TS de3327d4b00a4485b9f77b16b80b8365 (127.5.151.194:36181): tablet 7aa752eedaf546e88fb1f877bd994232 (table test-table [id=24444dffb81044cc9e73a910f05df7bb]) successfully deleted
I20260927 18:34:44.541980 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:44.541852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36311 (local address 127.0.0.1:55440)
0927 18:34:44.541913 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:44.541941 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 18:34:44.542062 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 12810
W20260927 18:34:44.542125 12607 catalog_manager.cc:4584] TS eb4d3886c1d544d984c4ba30ddb45751 (127.5.151.193:36311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7aa752eedaf546e88fb1f877bd994232: Network error: Client connection negotiation failed: client connection to 127.5.151.193:36311: connect: Connection refused (error 111)
I20260927 18:34:44.548076 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 12946
I20260927 18:34:44.553740 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 12302
I20260927 18:34:44.558250 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 12490
2026-09-27T18:34:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4696 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:44 dist-test-slave-19nl krb5kdc[13204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:44 dist-test-slave-19nl krb5kdc[13204](info): set up 2 sockets
Sep 27 18:34:44 dist-test-slave-19nl krb5kdc[13204](info): commencing operation
krb5kdc: starting...
W20260927 18:34:45.582864 5727 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:45 dist-test-slave-19nl krb5kdc[13204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab.
Entry for principal oryx/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab.
I20260927 18:34:45.689038 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43127
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab
--principal=oryx/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:43127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:45.765950 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:45.766139 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:45.766161 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:45.767385 13217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:45.767436 13217 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:34:45.767450 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:45.767463 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:45.767513 13217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:45.767529 13217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:45.768980 13217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:43127
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.254.keytab
--principal=oryx/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:45.769224 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:45.769459 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:45.774045 13225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:45.774123 13217 server_base.cc:1034] running on GCE node
W20260927 18:34:45.774051 13223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:45.774060 13222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:45.774423 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:45.774747 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:45.775887 13217 hybrid_clock.cc:648] HybridClock initialized: now 1790534085775869 us; error 27 us; skew 500 ppm
Sep 27 18:34:45 dist-test-slave-19nl krb5kdc[13204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:45.778240 13217 init.cc:377] Logged in from keytab as oryx/127.5.151.254@KRBTEST.COM (short username oryx)
I20260927 18:34:45.778702 13217 webserver.cc:460] Webserver started at http://127.5.151.254:40737/ using document root <none> and password file <none>
I20260927 18:34:45.778932 13217 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:45.779004 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:45.779155 13217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:45.780175 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "241fb61a681048fbac8e08ea0ae02309"
format_stamp: "Formatted at 2026-09-27 18:34:45 on dist-test-slave-19nl"
server_key: "ed55cf5529a1d8e761a74494727cad4f"
server_key_iv: "8839c344d0093706b980242ec8d4b0e9"
server_key_version: "encryptionkey@0"
I20260927 18:34:45.780514 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "241fb61a681048fbac8e08ea0ae02309"
format_stamp: "Formatted at 2026-09-27 18:34:45 on dist-test-slave-19nl"
server_key: "ed55cf5529a1d8e761a74494727cad4f"
server_key_iv: "8839c344d0093706b980242ec8d4b0e9"
server_key_version: "encryptionkey@0"
I20260927 18:34:45.781792 13217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:45.782603 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:45.782792 13217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:45.782895 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "241fb61a681048fbac8e08ea0ae02309"
format_stamp: "Formatted at 2026-09-27 18:34:45 on dist-test-slave-19nl"
server_key: "ed55cf5529a1d8e761a74494727cad4f"
server_key_iv: "8839c344d0093706b980242ec8d4b0e9"
server_key_version: "encryptionkey@0"
I20260927 18:34:45.782984 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:45.794761 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:45.796906 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:45.797065 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:45.801697 13217 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:43127
I20260927 18:34:45.801754 13294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:43127 every 8 connection(s)
I20260927 18:34:45.802060 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:45.802695 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:45.802924 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 13217
I20260927 18:34:45.803028 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:45.803231 5727 external_mini_cluster.cc:1428] Setting key c77fe57f038bf2cd4b8d6ebe58568765
I20260927 18:34:45.805280 13295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Bootstrap starting.
I20260927 18:34:45.805938 13295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:45.806289 13295 log.cc:824] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:45.807112 13295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: No bootstrap required, opened a new log
I20260927 18:34:45.809223 13295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241fb61a681048fbac8e08ea0ae02309" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43127 } }
I20260927 18:34:45.809355 13295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:45.809376 13295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241fb61a681048fbac8e08ea0ae02309, State: Initialized, Role: FOLLOWER
I20260927 18:34:45.809521 13295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241fb61a681048fbac8e08ea0ae02309" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43127 } }
I20260927 18:34:45.809592 13295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:45.809633 13295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:45.809679 13295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:45.810156 13295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241fb61a681048fbac8e08ea0ae02309" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43127 } }
I20260927 18:34:45.810235 13295 leader_election.cc:302] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241fb61a681048fbac8e08ea0ae02309; no voters:
I20260927 18:34:45.810416 13295 leader_election.cc:288] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:45.810441 13300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:45.810602 13300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [term 1 LEADER]: Becoming Leader. State: Replica: 241fb61a681048fbac8e08ea0ae02309, State: Running, Role: LEADER
I20260927 18:34:45.810705 13295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:45.810693 13300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241fb61a681048fbac8e08ea0ae02309" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43127 } }
I20260927 18:34:45.810978 13300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241fb61a681048fbac8e08ea0ae02309. Latest consensus state: current_term: 1 leader_uuid: "241fb61a681048fbac8e08ea0ae02309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241fb61a681048fbac8e08ea0ae02309" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43127 } } }
I20260927 18:34:45.810997 13302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241fb61a681048fbac8e08ea0ae02309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241fb61a681048fbac8e08ea0ae02309" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43127 } } }
I20260927 18:34:45.811071 13300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:45.811079 13302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:45.811254 13304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:45.811744 13304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 18:34:45 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:45.813424 13304 catalog_manager.cc:1347] Generated new cluster ID: b63b1e06606e4be4b8cd47ea42933060
I20260927 18:34:45.813499 13304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:45.815799 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:45.803871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.5.151.254:43127)
0927 18:34:45.804036 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:45.804045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:45.805564 (+ 1519us) server_negotiation.cc:407] Connection header received
0927 18:34:45.806006 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:45.806012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:45.806091 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:45.806223 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:45.807068 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.807963 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.808985 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.809377 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.809478 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:45.812334 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:45.812338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:45.812339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:45.812351 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:45.814211 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:45.814663 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:45.814666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:45.814666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:45.814759 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:45.814966 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:45.814967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:45.814967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:45.815196 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:45.815351 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:45.815463 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 18:34:45.815593 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260927 18:34:45.819811 13304 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:45.820274 13304 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:45.823561 13304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Generated new TSK 0
I20260927 18:34:45.823729 13304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab.
Entry for principal oryx/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab.
I20260927 18:34:45.851248 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43127
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab
--principal=oryx/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:45.928820 13321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:45.929047 13321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:45.929088 13321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:45.930234 13321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:45.930299 13321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:45.930335 13321 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:45.930392 13321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:45.931938 13321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab
--principal=oryx/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:45.932204 13321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:45.932437 13321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:45.937255 13326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:45.937264 13327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:45.937260 13329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:45.937536 13321 server_base.cc:1034] running on GCE node
I20260927 18:34:45.937716 13321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:45.937953 13321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:45.939090 13321 hybrid_clock.cc:648] HybridClock initialized: now 1790534085939068 us; error 36 us; skew 500 ppm
Sep 27 18:34:45 dist-test-slave-19nl krb5kdc[13204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:45.941596 13321 init.cc:377] Logged in from keytab as oryx/127.5.151.193@KRBTEST.COM (short username oryx)
I20260927 18:34:45.942029 13321 webserver.cc:460] Webserver started at http://127.5.151.193:41987/ using document root <none> and password file <none>
I20260927 18:34:45.942210 13321 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:45.942250 13321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:45.942338 13321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:45.943396 13321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "fd8d406a6eca426ab409fc982b3e38ff"
format_stamp: "Formatted at 2026-09-27 18:34:45 on dist-test-slave-19nl"
server_key: "81b3968fe827ad37f3d997e12a08658b"
server_key_iv: "8a81a7e1c16bf8496da1f20d1a2712b7"
server_key_version: "encryptionkey@0"
I20260927 18:34:45.943698 13321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "fd8d406a6eca426ab409fc982b3e38ff"
format_stamp: "Formatted at 2026-09-27 18:34:45 on dist-test-slave-19nl"
server_key: "81b3968fe827ad37f3d997e12a08658b"
server_key_iv: "8a81a7e1c16bf8496da1f20d1a2712b7"
server_key_version: "encryptionkey@0"
I20260927 18:34:45.944911 13321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:34:45.945582 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:45.945744 13321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:45.945825 13321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "fd8d406a6eca426ab409fc982b3e38ff"
format_stamp: "Formatted at 2026-09-27 18:34:45 on dist-test-slave-19nl"
server_key: "81b3968fe827ad37f3d997e12a08658b"
server_key_iv: "8a81a7e1c16bf8496da1f20d1a2712b7"
server_key_version: "encryptionkey@0"
I20260927 18:34:45.945878 13321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:45.957813 13321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:45.959542 13321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:45.959646 13321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:45.960052 13321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:45.960090 13321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:45.960111 13321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:45.960124 13321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:45.966627 13321 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:37305
I20260927 18:34:45.966696 13451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:37305 every 8 connection(s)
I20260927 18:34:45.966986 13321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:34:45 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:45.969856 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:45.960543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:47583 (local address 127.5.151.254:43127)
0927 18:34:45.960581 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:45.960583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:45.960779 (+ 196us) server_negotiation.cc:407] Connection header received
0927 18:34:45.961247 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:45.961248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:45.961263 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:45.961339 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:45.962179 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.963040 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.964586 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.964897 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.964985 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:45.967621 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:45.967622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:45.967622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:45.967629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:45.968746 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:45.969123 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:45.969123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:45.969124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:45.969176 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:45.969377 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:45.969377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:45.969377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:45.969478 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:45.969527 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:45.969726 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 18:34:45.969794 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:34:45.970083 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:45.960531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43127 (local address 127.5.151.193:47583)
0927 18:34:45.960673 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:45.960705 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:45.961144 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:45.961427 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:45.961435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:45.961786 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:45.962055 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:45.962062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.963131 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.963132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:45.964509 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:45.964513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.964996 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.964997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:45.965132 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:45.965708 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:45.965718 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:45.967525 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:45.968819 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:45.968825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:45.968827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:45.969054 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:45.969233 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:45.969234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:45.969234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:45.969324 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:45.969531 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:45.969535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:45.969639 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 18:34:45.969717 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:34:45.969830 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260927 18:34:45.970566 13452 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43127
I20260927 18:34:45.970650 13452 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:45.970824 13452 heartbeater.cc:511] Master 127.5.151.254:43127 requested a full tablet report, sending...
I20260927 18:34:45.971397 13248 ts_manager.cc:194] Registered new tserver with Master: fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305)
I20260927 18:34:45.972329 13248 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:47583
I20260927 18:34:45.976125 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 13321
I20260927 18:34:45.976244 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:45.976449 5727 external_mini_cluster.cc:1428] Setting key ab99bca5c20d871dd9f3bdcb00224fa1
I20260927 18:34:45.976971 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:45.971572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:45853 (local address 127.5.151.254:43127)
0927 18:34:45.971612 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:45.971615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:45.971733 (+ 118us) server_negotiation.cc:407] Connection header received
0927 18:34:45.971802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:45.971803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:45.971811 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:45.971851 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:45.972201 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.972769 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.973586 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.973759 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.973807 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:45.974590 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:45.974591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:45.974591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:45.974604 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:45.975901 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:45.976246 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:45.976248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:45.976248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:45.976307 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:45.976505 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:45.976506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:45.976506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:45.976616 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:45.976694 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:45.976854 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:34:45.976915 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:45.976996 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:45.971514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43127 (local address 127.5.151.193:45853)
0927 18:34:45.971674 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:45.971684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:45.971746 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:45.971868 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:45.971869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:45.972056 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:45.972112 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:45.972113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.972865 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.972867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:45.973515 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:45.973518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:45.973820 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:45.973821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:45.973866 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:45.974219 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:45.974223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:45.974482 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:45.975995 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:45.975996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:45.975996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:45.976147 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:45.976374 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:45.976375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:45.976376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:45.976440 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:45.976684 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:45.976685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:45.976730 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:34:45.976801 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:34:45.976899 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":25,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab.
Entry for principal oryx/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab.
I20260927 18:34:46.005460 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43127
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab
--principal=oryx/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:46.084828 13461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:46.085008 13461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:46.085031 13461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:46.086328 13461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:46.086370 13461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:46.086390 13461 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:46.086432 13461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:46.087945 13461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab
--principal=oryx/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:46.088181 13461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:46.088402 13461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:46.093127 13466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:46.093117 13469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:46.093117 13467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:46.093355 13461 server_base.cc:1034] running on GCE node
I20260927 18:34:46.093541 13461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:46.093722 13461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:46.094888 13461 hybrid_clock.cc:648] HybridClock initialized: now 1790534086094877 us; error 24 us; skew 500 ppm
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:46.097160 13461 init.cc:377] Logged in from keytab as oryx/127.5.151.194@KRBTEST.COM (short username oryx)
I20260927 18:34:46.097594 13461 webserver.cc:460] Webserver started at http://127.5.151.194:44329/ using document root <none> and password file <none>
I20260927 18:34:46.097805 13461 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:46.097870 13461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:46.098014 13461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:46.099063 13461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "18bbfe20b5ab46a68801b3f800f4b6cf"
format_stamp: "Formatted at 2026-09-27 18:34:46 on dist-test-slave-19nl"
server_key: "058a32714c7a0f23eb7c7b727314ec8b"
server_key_iv: "ec8cfbc11c55871244ff27afeb198b3a"
server_key_version: "encryptionkey@0"
I20260927 18:34:46.099416 13461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "18bbfe20b5ab46a68801b3f800f4b6cf"
format_stamp: "Formatted at 2026-09-27 18:34:46 on dist-test-slave-19nl"
server_key: "058a32714c7a0f23eb7c7b727314ec8b"
server_key_iv: "ec8cfbc11c55871244ff27afeb198b3a"
server_key_version: "encryptionkey@0"
I20260927 18:34:46.100708 13461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:46.101372 13476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.101522 13461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:46.101616 13461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "18bbfe20b5ab46a68801b3f800f4b6cf"
format_stamp: "Formatted at 2026-09-27 18:34:46 on dist-test-slave-19nl"
server_key: "058a32714c7a0f23eb7c7b727314ec8b"
server_key_iv: "ec8cfbc11c55871244ff27afeb198b3a"
server_key_version: "encryptionkey@0"
I20260927 18:34:46.101693 13461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:46.120066 13461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:46.121876 13461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:46.122042 13461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:46.122637 13461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:46.122695 13461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.122728 13461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:46.122753 13461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:46.129927 13461 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:34043
I20260927 18:34:46.130354 13461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:46.130635 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 13461
I20260927 18:34:46.130767 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:46.130951 5727 external_mini_cluster.cc:1428] Setting key 2fa0185b66502509c1565158593ec6a1
I20260927 18:34:46.130327 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:34043 every 8 connection(s)
I20260927 18:34:46.132596 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.123010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:58179 (local address 127.5.151.254:43127)
0927 18:34:46.123078 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.123080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.123401 (+ 321us) server_negotiation.cc:407] Connection header received
0927 18:34:46.124141 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.124142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.124152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.124187 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.125112 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.125595 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.126578 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.126788 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.126885 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.129419 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:46.129421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:46.129421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:46.129430 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:46.131101 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.131629 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.131630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.131631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.131696 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.131950 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.131951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.131951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.132074 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:46.132144 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.132421 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.132504 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:34:46.132831 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.123015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43127 (local address 127.5.151.194:58179)
0927 18:34:46.123227 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.123280 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.124019 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.124278 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.124282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.124714 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:46.125010 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.125017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.125688 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.125690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.126461 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.126464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.126896 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.126897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.127064 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.127925 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:46.127933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:46.129324 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:46.131193 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.131199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.131201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.131550 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.131764 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.131765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.131766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.131871 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.132147 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:46.132154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:46.132290 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.132400 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.132566 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260927 18:34:46.133276 13592 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43127
I20260927 18:34:46.133350 13592 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:46.133544 13592 heartbeater.cc:511] Master 127.5.151.254:43127 requested a full tablet report, sending...
I20260927 18:34:46.133895 13249 ts_manager.cc:194] Registered new tserver with Master: 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043)
I20260927 18:34:46.134367 13249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:58179
WARNING: no policy specified for oryx/127.5.151.195@KRBTEST.COM; defaulting to no policy
I20260927 18:34:46.139955 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.134297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43127 (local address 127.5.151.194:60087)
0927 18:34:46.134417 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.134427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.134497 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.134689 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.134690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.134899 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:46.134969 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.134970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.135616 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.135617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.136104 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.136105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.136476 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.136477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.136551 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.136975 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:46.136980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:46.137250 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:46.138865 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.138866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.138867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.139065 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.139257 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.139258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.139258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.139315 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.139620 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:46.139621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:46.139666 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.139753 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.139854 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260927 18:34:46.140058 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.134351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:60087 (local address 127.5.151.254:43127)
0927 18:34:46.134385 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.134387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.134473 (+ 86us) server_negotiation.cc:407] Connection header received
0927 18:34:46.134580 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.134581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.134593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.134630 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.135054 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.135523 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.136198 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.136396 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.136475 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.137351 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:46.137352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:46.137353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:46.137359 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:46.138776 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.139134 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.139135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.139135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.139193 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.139401 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.139402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.139403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.139549 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:46.139628 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.139780 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.139857 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab.
Entry for principal oryx/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab.
I20260927 18:34:46.154400 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43127
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab
--principal=oryx/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:46.232229 13601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:46.232415 13601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:46.232456 13601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:46.233563 13601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:46.233599 13601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:46.233621 13601 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:34:46.233639 13601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:46.235133 13601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab
--principal=oryx/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.151.254:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:46.235352 13601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:46.235618 13601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:46.240446 13607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:46.240453 13606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:46.240454 13609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:46.240628 13601 server_base.cc:1034] running on GCE node
I20260927 18:34:46.240836 13601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:46.241067 13601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:46.242244 13601 hybrid_clock.cc:648] HybridClock initialized: now 1790534086242225 us; error 32 us; skew 500 ppm
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:46.244585 13601 init.cc:377] Logged in from keytab as oryx/127.5.151.195@KRBTEST.COM (short username oryx)
I20260927 18:34:46.245038 13601 webserver.cc:460] Webserver started at http://127.5.151.195:32815/ using document root <none> and password file <none>
I20260927 18:34:46.245271 13601 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:46.245337 13601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:46.245465 13601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:46.246423 13601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "f454bb5d7ff940a09cd681047982a3b0"
format_stamp: "Formatted at 2026-09-27 18:34:46 on dist-test-slave-19nl"
server_key: "47d20d56c7308667425792202edfd4d1"
server_key_iv: "90b195c6654c68ee300010852b50a651"
server_key_version: "encryptionkey@0"
I20260927 18:34:46.246780 13601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "f454bb5d7ff940a09cd681047982a3b0"
format_stamp: "Formatted at 2026-09-27 18:34:46 on dist-test-slave-19nl"
server_key: "47d20d56c7308667425792202edfd4d1"
server_key_iv: "90b195c6654c68ee300010852b50a651"
server_key_version: "encryptionkey@0"
I20260927 18:34:46.248152 13601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:46.248868 13616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.249042 13601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:46.249143 13601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "f454bb5d7ff940a09cd681047982a3b0"
format_stamp: "Formatted at 2026-09-27 18:34:46 on dist-test-slave-19nl"
server_key: "47d20d56c7308667425792202edfd4d1"
server_key_iv: "90b195c6654c68ee300010852b50a651"
server_key_version: "encryptionkey@0"
I20260927 18:34:46.249228 13601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:46.266147 13601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:46.267908 13601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:46.268064 13601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:46.268651 13601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:46.268718 13601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.268747 13601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:46.268765 13601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:46.275870 13601 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:35437
I20260927 18:34:46.275945 13731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:35437 every 8 connection(s)
I20260927 18:34:46.276279 13601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:34:46.278590 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 13601
I20260927 18:34:46.278628 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.268983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:50735 (local address 127.5.151.254:43127)
0927 18:34:46.269048 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.269050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.269209 (+ 159us) server_negotiation.cc:407] Connection header received
0927 18:34:46.269700 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.269702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.269715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.269777 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.270406 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.271268 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.272974 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.273310 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.273385 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.276072 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:46.276074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:46.276074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:46.276083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:46.277431 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.277789 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.277790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.277790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.277853 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.278062 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.278062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.278063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.278155 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:46.278203 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.278428 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.278503 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:34:46.278700 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:46.278755 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.268945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43127 (local address 127.5.151.195:50735)
0927 18:34:46.269090 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.269134 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.269598 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.269840 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.269843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.270110 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:46.270313 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.270317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.271384 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.271385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.272881 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.272884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.273421 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.273423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.273591 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.274238 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:46.274249 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:46.275969 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:46.277501 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.277505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.277507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.277704 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.277909 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.277909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.277910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.278008 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.278206 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:46.278210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:46.278307 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.278379 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.278501 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260927 18:34:46.278934 5727 external_mini_cluster.cc:1428] Setting key 6df8277ced1aac4d687db80a04f5fefb
I20260927 18:34:46.279428 13732 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43127
I20260927 18:34:46.279522 13732 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:46.279683 13732 heartbeater.cc:511] Master 127.5.151.254:43127 requested a full tablet report, sending...
I20260927 18:34:46.280031 13247 ts_manager.cc:194] Registered new tserver with Master: f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437)
I20260927 18:34:46.280464 13247 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:50735
I20260927 18:34:46.284761 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.280351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:60137 (local address 127.5.151.254:43127)
0927 18:34:46.280384 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.280386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.280444 (+ 58us) server_negotiation.cc:407] Connection header received
0927 18:34:46.280513 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.280514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.280523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.280573 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.280889 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.281303 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.281934 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.282118 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.282171 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.282990 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:46.282990 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:46.282991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:46.282996 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:46.283992 (+ 996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.284272 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.284272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.284273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.284314 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.284449 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.284450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.284450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.284525 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:46.284569 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.284667 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.284713 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:46.284881 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.280297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43127 (local address 127.5.151.195:60137)
0927 18:34:46.280395 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.280405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.280459 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.280591 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.280591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.280742 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:46.280812 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.280813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.281390 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.281391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.281861 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.281863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.282183 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.282184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.282228 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.282659 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:46.282664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:46.282914 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:46.284069 (+ 1155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.284070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.284070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.284216 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.284368 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.284368 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.284369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.284413 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.284573 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:46.284574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:46.284604 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.284652 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.284757 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":59,"threads_started":1}
I20260927 18:34:46.290504 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:34:46.310303 13298 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:46.305408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.5.151.254:43127)
0927 18:34:46.305459 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.305461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.305516 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:34:46.305566 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.305567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.305575 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.305614 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.306123 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.306620 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.307286 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.307451 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.307500 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.310254 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41550: BlockingRecv error: recv got EOF from 127.0.0.1:41550 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:46.317523 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.311186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.5.151.254:43127)
0927 18:34:46.311268 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.311269 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.311331 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:34:46.311383 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.311384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.311390 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.311444 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.311869 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.312418 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.313069 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.313229 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.313280 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.315300 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:46.315301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:46.315302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:46.315309 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:46.316461 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.316818 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.316819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.316820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.316875 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.317065 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.317065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.317066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.317166 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:46.317261 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.317361 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.317429 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:34:46.319023 13247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:46.320539 13247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:46.326849 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.322544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35437 (local address 127.0.0.1:56108)
0927 18:34:46.322838 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.322877 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.322944 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.323319 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.323320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.323322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.323378 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.323380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.324597 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.324598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.325609 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.325611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.326598 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.326599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.326657 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.326660 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.326708 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.326769 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":83,"threads_started":1}
I20260927 18:34:46.326860 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.322719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.5.151.193:37305)
0927 18:34:46.322898 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.322903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.323018 (+ 115us) server_negotiation.cc:407] Connection header received
0927 18:34:46.323152 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.323154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.323167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.323251 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.323423 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.324291 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.325939 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.326443 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.326513 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.326615 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.326722 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.326760 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260927 18:34:46.326869 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.322620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:37305 (local address 127.0.0.1:50666)
0927 18:34:46.322934 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.322942 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.323047 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.323274 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.323278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.323279 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.323341 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.323342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.324369 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.324370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.325867 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.325868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.326514 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.326515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.326634 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.326637 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.326709 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.326781 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":26,"thread_start_us":47,"threads_started":1}
I20260927 18:34:46.327071 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.322771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:34043 (local address 127.0.0.1:48280)
0927 18:34:46.323327 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.323337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.323383 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.323580 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.323581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.323582 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.323627 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.323628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.324765 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.324766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.325488 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.325490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.326394 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.326395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.326476 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.326494 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.326967 (+ 473us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.327009 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":34,"threads_started":1}
I20260927 18:34:46.327301 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.322869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.5.151.194:34043)
0927 18:34:46.323060 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.323064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.323438 (+ 374us) server_negotiation.cc:407] Connection header received
0927 18:34:46.323478 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.323480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.323498 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.323570 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.323693 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.324673 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.325595 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.326309 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.326383 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.326455 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.327155 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.327205 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260927 18:34:46.327826 13386 tablet_service.cc:1467] Processing CreateTablet for tablet 217e0fa38030455dbc7d9563658dafe3 (DEFAULT_TABLE table=test-table [id=a3bfb86dfc5a4c2a9774e70ca16962c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.327874 13526 tablet_service.cc:1467] Processing CreateTablet for tablet 217e0fa38030455dbc7d9563658dafe3 (DEFAULT_TABLE table=test-table [id=a3bfb86dfc5a4c2a9774e70ca16962c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.328172 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217e0fa38030455dbc7d9563658dafe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.328173 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217e0fa38030455dbc7d9563658dafe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.326869 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.322668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.5.151.195:35437)
0927 18:34:46.323102 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.323106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.323116 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:46.323150 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.323152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.323167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.323255 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.323458 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.324497 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.325678 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.326317 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.326383 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.326456 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.326735 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.326774 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":81,"threads_started":1}
I20260927 18:34:46.331118 13666 tablet_service.cc:1467] Processing CreateTablet for tablet 217e0fa38030455dbc7d9563658dafe3 (DEFAULT_TABLE table=test-table [id=a3bfb86dfc5a4c2a9774e70ca16962c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.331451 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217e0fa38030455dbc7d9563658dafe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.332140 13757 tablet_bootstrap.cc:492] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Bootstrap starting.
I20260927 18:34:46.332387 13758 tablet_bootstrap.cc:492] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Bootstrap starting.
I20260927 18:34:46.332804 13757 tablet_bootstrap.cc:654] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.332973 13758 tablet_bootstrap.cc:654] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.333174 13757 log.cc:824] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:46.333330 13758 log.cc:824] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:46.333971 13757 tablet_bootstrap.cc:492] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: No bootstrap required, opened a new log
I20260927 18:34:46.334041 13757 ts_tablet_manager.cc:1397] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:46.334115 13758 tablet_bootstrap.cc:492] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: No bootstrap required, opened a new log
I20260927 18:34:46.334194 13758 ts_tablet_manager.cc:1397] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:46.335031 13761 tablet_bootstrap.cc:492] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Bootstrap starting.
I20260927 18:34:46.335594 13761 tablet_bootstrap.cc:654] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.335892 13761 log.cc:824] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:46.335896 13757 raft_consensus.cc:348] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.335919 13758 raft_consensus.cc:348] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.336050 13757 raft_consensus.cc:374] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.336093 13757 raft_consensus.cc:729] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bbfe20b5ab46a68801b3f800f4b6cf, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.336088 13758 raft_consensus.cc:374] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.336129 13758 raft_consensus.cc:729] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8d406a6eca426ab409fc982b3e38ff, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.336199 13757 consensus_queue.cc:260] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.336237 13758 consensus_queue.cc:260] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.336467 13757 ts_tablet_manager.cc:1428] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:34:46.336503 13758 ts_tablet_manager.cc:1428] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:34:46.336514 13452 heartbeater.cc:503] Master 127.5.151.254:43127 was elected leader, sending a full tablet report...
I20260927 18:34:46.336706 13592 heartbeater.cc:503] Master 127.5.151.254:43127 was elected leader, sending a full tablet report...
I20260927 18:34:46.337574 13761 tablet_bootstrap.cc:492] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: No bootstrap required, opened a new log
I20260927 18:34:46.337648 13761 ts_tablet_manager.cc:1397] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:34:46.339128 13761 raft_consensus.cc:348] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.339242 13761 raft_consensus.cc:374] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.339262 13761 raft_consensus.cc:729] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f454bb5d7ff940a09cd681047982a3b0, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.339361 13761 consensus_queue.cc:260] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.339563 13761 ts_tablet_manager.cc:1428] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:34:46.339630 13732 heartbeater.cc:503] Master 127.5.151.254:43127 was elected leader, sending a full tablet report...
W20260927 18:34:46.381405 13593 tablet.cc:2369] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:46.440196 13764 raft_consensus.cc:482] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:46.440358 13764 raft_consensus.cc:504] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.440721 13764 leader_election.cc:288] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437), 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043)
I20260927 18:34:46.443483 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.441026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:55889 (local address 127.5.151.195:35437)
0927 18:34:46.441078 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.441081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.441093 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:34:46.441121 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.441122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.441131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.441184 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.441336 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.441795 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.442742 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.443100 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.443184 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.443194 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.443393 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.443427 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:34:46.443483 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.440904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35437 (local address 127.5.151.193:55889)
0927 18:34:46.440969 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.440981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.441056 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.441190 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.441191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.441193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.441264 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.441265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.441877 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.441878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.442657 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.442659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.443181 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.443182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.443296 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.443299 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.443374 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.443414 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 18:34:46.443789 13686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217e0fa38030455dbc7d9563658dafe3" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f454bb5d7ff940a09cd681047982a3b0" is_pre_election: true
I20260927 18:34:46.443899 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.441020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:34043 (local address 127.5.151.193:58239)
0927 18:34:46.441183 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.441192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.441241 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.441386 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.441387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.441388 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.441466 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.441468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.442147 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.442149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.443031 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.443034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.443658 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.443659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.443722 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.443726 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.443780 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.443819 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
I20260927 18:34:46.443903 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.441119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:58239 (local address 127.5.151.194:34043)
0927 18:34:46.441166 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.441168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.441251 (+ 83us) server_negotiation.cc:407] Connection header received
0927 18:34:46.441299 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.441299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.441307 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.441369 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.441527 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.442018 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.443149 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.443575 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.443647 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.443657 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.443793 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.443833 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 18:34:46.443939 13686 raft_consensus.cc:2454] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 0.
I20260927 18:34:46.444195 13339 leader_election.cc:302] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f454bb5d7ff940a09cd681047982a3b0, fd8d406a6eca426ab409fc982b3e38ff; no voters:
I20260927 18:34:46.444326 13764 raft_consensus.cc:2781] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:46.444378 13764 raft_consensus.cc:482] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:46.444327 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217e0fa38030455dbc7d9563658dafe3" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" is_pre_election: true
I20260927 18:34:46.444396 13764 raft_consensus.cc:3037] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.444473 13546 raft_consensus.cc:2454] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 0.
I20260927 18:34:46.445363 13764 raft_consensus.cc:504] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.445497 13764 leader_election.cc:288] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 election: Requested vote from peers f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437), 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043)
I20260927 18:34:46.445653 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217e0fa38030455dbc7d9563658dafe3" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf"
I20260927 18:34:46.445685 13686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217e0fa38030455dbc7d9563658dafe3" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f454bb5d7ff940a09cd681047982a3b0"
I20260927 18:34:46.445748 13546 raft_consensus.cc:3037] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.445749 13686 raft_consensus.cc:3037] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.446401 13546 raft_consensus.cc:2454] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 1.
I20260927 18:34:46.446412 13686 raft_consensus.cc:2454] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 1.
I20260927 18:34:46.446576 13339 leader_election.cc:302] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f454bb5d7ff940a09cd681047982a3b0, fd8d406a6eca426ab409fc982b3e38ff; no voters:
I20260927 18:34:46.446655 13764 raft_consensus.cc:2781] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:46.446758 13764 raft_consensus.cc:686] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 1 LEADER]: Becoming Leader. State: Replica: fd8d406a6eca426ab409fc982b3e38ff, State: Running, Role: LEADER
I20260927 18:34:46.446848 13764 consensus_queue.cc:237] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } }
I20260927 18:34:46.447629 13248 catalog_manager.cc:5495] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193). New cstate: current_term: 1 leader_uuid: "fd8d406a6eca426ab409fc982b3e38ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } health_report { overall_health: HEALTHY } } }
W20260927 18:34:46.467741 13453 tablet.cc:2369] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:46.469911 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.467720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.5.151.193:37305)
0927 18:34:46.467782 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.467784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.467793 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:46.467829 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.467829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.467847 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.467931 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:46.468082 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.468547 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.469244 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.469421 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.469481 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.469618 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.469707 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.469762 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.469799 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.469817 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:34:46.471737 13546 raft_consensus.cc:1261] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Refusing update from remote peer fd8d406a6eca426ab409fc982b3e38ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:46.471757 13686 raft_consensus.cc:1261] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Refusing update from remote peer fd8d406a6eca426ab409fc982b3e38ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:46.472108 13764 consensus_queue.cc:1035] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:46.472259 13767 consensus_queue.cc:1035] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:46.474712 13772 mvcc.cc:205] Tried to move back new op lower bound from 7334027618185568256 to 7334027618086604800. Current Snapshot: MvccSnapshot[applied={T|T < 7334027618185568256}]
I20260927 18:34:46.475266 13771 mvcc.cc:205] Tried to move back new op lower bound from 7334027618185568256 to 7334027618086604800. Current Snapshot: MvccSnapshot[applied={T|T < 7334027618185568256}]
I20260927 18:34:46.476540 13248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:46.476629 13248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:46.477424 13248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Sending DeleteTablet for 3 replicas of tablet 217e0fa38030455dbc7d9563658dafe3
I20260927 18:34:46.477787 13666 tablet_service.cc:1514] Processing DeleteTablet for tablet 217e0fa38030455dbc7d9563658dafe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:46 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:56108
I20260927 18:34:46.477797 13386 tablet_service.cc:1514] Processing DeleteTablet for tablet 217e0fa38030455dbc7d9563658dafe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:46 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50666
I20260927 18:34:46.477931 13526 tablet_service.cc:1514] Processing DeleteTablet for tablet 217e0fa38030455dbc7d9563658dafe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:46 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:48280
I20260927 18:34:46.478051 13780 tablet_replica.cc:330] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: stopping tablet replica
I20260927 18:34:46.478142 13780 raft_consensus.cc:2229] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:46.478134 13779 tablet_replica.cc:330] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: stopping tablet replica
I20260927 18:34:46.478233 13779 raft_consensus.cc:2229] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:46.478322 13779 raft_consensus.cc:2258] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:46.478374 13248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:46.478513 13248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:46.478657 13780 raft_consensus.cc:2258] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:46.478791 13779 ts_tablet_manager.cc:1905] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:46.479064 13780 ts_tablet_manager.cc:1905] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:46.480244 13526 tablet_service.cc:1467] Processing CreateTablet for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c (DEFAULT_TABLE table=test-table [id=3e12cca25a3745ebbe7eadf59ae37e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.480368 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.480686 13780 ts_tablet_manager.cc:1918] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:46.480693 13779 ts_tablet_manager.cc:1918] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:46.480744 13780 log.cc:1196] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/217e0fa38030455dbc7d9563658dafe3
I20260927 18:34:46.480753 13779 log.cc:1196] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/217e0fa38030455dbc7d9563658dafe3
I20260927 18:34:46.480998 13780 ts_tablet_manager.cc:1939] T 217e0fa38030455dbc7d9563658dafe3 P f454bb5d7ff940a09cd681047982a3b0: Deleting consensus metadata
I20260927 18:34:46.480998 13779 ts_tablet_manager.cc:1939] T 217e0fa38030455dbc7d9563658dafe3 P fd8d406a6eca426ab409fc982b3e38ff: Deleting consensus metadata
I20260927 18:34:46.481716 13757 tablet_bootstrap.cc:492] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: Bootstrap starting.
I20260927 18:34:46.481870 13235 catalog_manager.cc:4841] TS f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437): tablet 217e0fa38030455dbc7d9563658dafe3 (table test-table [id=a3bfb86dfc5a4c2a9774e70ca16962c7]) successfully deleted
I20260927 18:34:46.482016 13666 tablet_service.cc:1467] Processing CreateTablet for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c (DEFAULT_TABLE table=test-table [id=3e12cca25a3745ebbe7eadf59ae37e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.482101 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.482139 13757 tablet_bootstrap.cc:654] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.482767 13234 catalog_manager.cc:4841] TS fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305): tablet 217e0fa38030455dbc7d9563658dafe3 (table test-table [id=a3bfb86dfc5a4c2a9774e70ca16962c7]) successfully deleted
I20260927 18:34:46.482822 13757 tablet_bootstrap.cc:492] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: No bootstrap required, opened a new log
I20260927 18:34:46.482862 13757 ts_tablet_manager.cc:1397] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:46.482939 13386 tablet_service.cc:1467] Processing CreateTablet for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c (DEFAULT_TABLE table=test-table [id=3e12cca25a3745ebbe7eadf59ae37e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.483047 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.483011 13757 raft_consensus.cc:348] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.483084 13757 raft_consensus.cc:374] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.483107 13757 raft_consensus.cc:729] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bbfe20b5ab46a68801b3f800f4b6cf, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.483166 13757 consensus_queue.cc:260] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.483193 13781 tablet_replica.cc:330] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: stopping tablet replica
I20260927 18:34:46.483259 13757 ts_tablet_manager.cc:1428] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.483342 13781 raft_consensus.cc:2229] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:46.483392 13761 tablet_bootstrap.cc:492] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Bootstrap starting.
I20260927 18:34:46.483479 13781 raft_consensus.cc:2258] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:46.483778 13761 tablet_bootstrap.cc:654] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.483959 13781 ts_tablet_manager.cc:1905] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:46.484256 13758 tablet_bootstrap.cc:492] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: Bootstrap starting.
I20260927 18:34:46.484632 13758 tablet_bootstrap.cc:654] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.485051 13761 tablet_bootstrap.cc:492] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: No bootstrap required, opened a new log
I20260927 18:34:46.485101 13761 ts_tablet_manager.cc:1397] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:46.485241 13761 raft_consensus.cc:348] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.485309 13761 raft_consensus.cc:374] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.485337 13761 raft_consensus.cc:729] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f454bb5d7ff940a09cd681047982a3b0, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.485378 13761 consensus_queue.cc:260] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.485450 13761 ts_tablet_manager.cc:1428] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.485503 13781 ts_tablet_manager.cc:1918] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:46.485569 13781 log.cc:1196] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/217e0fa38030455dbc7d9563658dafe3
I20260927 18:34:46.485818 13781 ts_tablet_manager.cc:1939] T 217e0fa38030455dbc7d9563658dafe3 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting consensus metadata
I20260927 18:34:46.486013 13758 tablet_bootstrap.cc:492] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: No bootstrap required, opened a new log
I20260927 18:34:46.486065 13758 ts_tablet_manager.cc:1397] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:34:46.486214 13235 catalog_manager.cc:4841] TS 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043): tablet 217e0fa38030455dbc7d9563658dafe3 (table test-table [id=a3bfb86dfc5a4c2a9774e70ca16962c7]) successfully deleted
I20260927 18:34:46.486202 13758 raft_consensus.cc:348] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.486263 13758 raft_consensus.cc:374] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.486282 13758 raft_consensus.cc:729] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8d406a6eca426ab409fc982b3e38ff, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.486331 13758 consensus_queue.cc:260] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.486418 13758 ts_tablet_manager.cc:1428] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 18:34:46.526868 13733 tablet.cc:2369] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:34:46.729378 13765 raft_consensus.cc:482] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:46.729488 13765 raft_consensus.cc:504] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.729923 13765 leader_election.cc:288] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305), 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043)
I20260927 18:34:46.733045 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.730260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:43903 (local address 127.5.151.194:34043)
0927 18:34:46.730307 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.730308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.730524 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:34:46.730576 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.730577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.730587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.730656 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.730860 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.731348 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.732319 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.732681 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.732746 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.732754 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.732928 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.732972 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:34:46.733091 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.730164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:37305 (local address 127.5.151.195:35499)
0927 18:34:46.730239 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.730252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.730309 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.730546 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.730570 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.730572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.730652 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.730655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.731330 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.731332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.732172 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.732174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.732808 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.732810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.732874 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.732877 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.732964 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.732997 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 18:34:46.733106 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.730219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:35499 (local address 127.5.151.193:37305)
0927 18:34:46.730292 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.730294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.730307 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:34:46.730381 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.730382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.730395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.730468 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.730728 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.731255 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.732268 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.732699 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.732796 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.732804 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.732969 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.733010 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:34:46.733103 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.730168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:34043 (local address 127.5.151.195:43903)
0927 18:34:46.730409 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.730419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.730476 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.730676 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.730677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.730678 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.730782 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.730784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.731433 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.731434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.732206 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.732208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.732781 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.732783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.732848 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.732851 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.732903 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.732955 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":93,"thread_start_us":78,"threads_started":1}
I20260927 18:34:46.733383 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1592d4cb8a8f4409baaa4f05a1eb3f6c" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8d406a6eca426ab409fc982b3e38ff" is_pre_election: true
I20260927 18:34:46.733491 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1592d4cb8a8f4409baaa4f05a1eb3f6c" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" is_pre_election: true
I20260927 18:34:46.733598 13406 raft_consensus.cc:2454] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 0.
I20260927 18:34:46.733676 13546 raft_consensus.cc:2454] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 0.
I20260927 18:34:46.733834 13618 leader_election.cc:302] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f454bb5d7ff940a09cd681047982a3b0, fd8d406a6eca426ab409fc982b3e38ff; no voters:
I20260927 18:34:46.733999 13765 raft_consensus.cc:2781] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:46.734056 13765 raft_consensus.cc:482] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:46.734089 13765 raft_consensus.cc:3037] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.734869 13765 raft_consensus.cc:504] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.735025 13765 leader_election.cc:288] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 election: Requested vote from peers fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305), 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043)
I20260927 18:34:46.735159 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1592d4cb8a8f4409baaa4f05a1eb3f6c" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf"
I20260927 18:34:46.735186 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1592d4cb8a8f4409baaa4f05a1eb3f6c" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8d406a6eca426ab409fc982b3e38ff"
I20260927 18:34:46.735241 13546 raft_consensus.cc:3037] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.735252 13406 raft_consensus.cc:3037] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.735978 13546 raft_consensus.cc:2454] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 1.
I20260927 18:34:46.736109 13406 raft_consensus.cc:2454] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 1.
I20260927 18:34:46.736189 13619 leader_election.cc:302] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bbfe20b5ab46a68801b3f800f4b6cf, f454bb5d7ff940a09cd681047982a3b0; no voters:
I20260927 18:34:46.736284 13765 raft_consensus.cc:2781] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:46.736461 13765 raft_consensus.cc:686] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 1 LEADER]: Becoming Leader. State: Replica: f454bb5d7ff940a09cd681047982a3b0, State: Running, Role: LEADER
I20260927 18:34:46.736552 13765 consensus_queue.cc:237] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.737262 13248 catalog_manager.cc:5495] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "f454bb5d7ff940a09cd681047982a3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } health_report { overall_health: HEALTHY } } }
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.254@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:46.793471 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.783400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.5.151.254:43127)
0927 18:34:46.783491 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.783494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.783613 (+ 119us) server_negotiation.cc:407] Connection header received
0927 18:34:46.783730 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.783731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.783742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.783806 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.784310 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.784802 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.785583 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.785767 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.785849 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.791166 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:46.791167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:46.791168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:46.791174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:46.792287 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.792664 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.792665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.792666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.792720 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.792901 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.792902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.792902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.793016 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:46.793090 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.793234 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.793361 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:34:46.793560 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.783305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43127 (local address 127.0.0.1:41560)
0927 18:34:46.783534 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.783544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.783634 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.783810 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.783811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.784163 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:46.784214 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.784217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.784886 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.784887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.785477 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.785479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.785849 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.785850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.785916 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.789714 (+ 3798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:46.789724 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:46.791075 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:46.792382 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.792390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.792392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.792578 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.792783 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:46.792784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:46.792785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:46.792841 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:46.793080 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:46.793087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:46.793131 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.793213 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.793422 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":103,"thread_start_us":101,"threads_started":1}
I20260927 18:34:46.795418 13248 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:41560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:34:46.795663 13248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:46.797011 13386 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8471e726a742a889c6ba66c0d2c2e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ac0dece30b4d78b2f53d156164104e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:46.797048 13666 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8471e726a742a889c6ba66c0d2c2e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ac0dece30b4d78b2f53d156164104e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:46.797041 13526 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8471e726a742a889c6ba66c0d2c2e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ac0dece30b4d78b2f53d156164104e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:34:46.797168 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8471e726a742a889c6ba66c0d2c2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.797181 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8471e726a742a889c6ba66c0d2c2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.797181 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8471e726a742a889c6ba66c0d2c2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.798265 13758 tablet_bootstrap.cc:492] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: Bootstrap starting.
I20260927 18:34:46.798645 13758 tablet_bootstrap.cc:654] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.798643 13761 tablet_bootstrap.cc:492] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: Bootstrap starting.
I20260927 18:34:46.799010 13761 tablet_bootstrap.cc:654] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.799268 13758 tablet_bootstrap.cc:492] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: No bootstrap required, opened a new log
I20260927 18:34:46.799309 13758 ts_tablet_manager.cc:1397] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:46.799345 13757 tablet_bootstrap.cc:492] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: Bootstrap starting.
I20260927 18:34:46.799413 13758 raft_consensus.cc:348] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.799484 13758 raft_consensus.cc:374] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.799510 13758 raft_consensus.cc:729] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8d406a6eca426ab409fc982b3e38ff, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.799561 13758 consensus_queue.cc:260] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.799628 13761 tablet_bootstrap.cc:492] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: No bootstrap required, opened a new log
I20260927 18:34:46.799659 13758 ts_tablet_manager.cc:1428] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.799674 13761 ts_tablet_manager.cc:1397] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:46.799705 13757 tablet_bootstrap.cc:654] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.799678 13767 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.799774 13767 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:46.799815 13761 raft_consensus.cc:348] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.799871 13761 raft_consensus.cc:374] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.799889 13761 raft_consensus.cc:729] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f454bb5d7ff940a09cd681047982a3b0, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.799933 13761 consensus_queue.cc:260] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.800019 13761 ts_tablet_manager.cc:1428] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:46.800122 13765 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.800201 13765 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:46.800696 13757 tablet_bootstrap.cc:492] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: No bootstrap required, opened a new log
I20260927 18:34:46.800751 13757 ts_tablet_manager.cc:1397] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:46.800884 13757 raft_consensus.cc:348] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.800949 13757 raft_consensus.cc:374] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.800963 13757 raft_consensus.cc:729] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bbfe20b5ab46a68801b3f800f4b6cf, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.801014 13757 consensus_queue.cc:260] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.801118 13757 ts_tablet_manager.cc:1428] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.801151 13763 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.801259 13763 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:46.810415 13763 raft_consensus.cc:482] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:46.810606 13763 raft_consensus.cc:504] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.810902 13763 leader_election.cc:288] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305), f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437)
I20260927 18:34:46.813848 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.811196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:59993 (local address 127.5.151.195:35437)
0927 18:34:46.811248 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.811250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.811304 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:34:46.811380 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.811383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.811395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.811462 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.811650 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.812114 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.813089 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.813516 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.813567 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.813574 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.813729 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.813783 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:34:46.813865 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.811041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35437 (local address 127.5.151.194:59993)
0927 18:34:46.811201 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.811252 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.811318 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.811475 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.811476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.811479 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.811590 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.811593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.812209 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.812210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.813013 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.813016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.813586 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.813587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.813655 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.813659 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.813720 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.813767 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260927 18:34:46.814079 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.811269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:48029 (local address 127.5.151.193:37305)
0927 18:34:46.811309 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.811311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.811474 (+ 163us) server_negotiation.cc:407] Connection header received
0927 18:34:46.811529 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.811530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.811538 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.811606 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:34:46.811793 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.812284 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.813287 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.813707 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.813767 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.813775 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.813960 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.814011 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:34:46.814085 13686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8471e726a742a889c6ba66c0d2c2e6" candidate_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f454bb5d7ff940a09cd681047982a3b0" is_pre_election: true
I20260927 18:34:46.814178 13686 raft_consensus.cc:2454] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18bbfe20b5ab46a68801b3f800f4b6cf in term 0.
I20260927 18:34:46.814321 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.811054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:37305 (local address 127.5.151.194:48029)
0927 18:34:46.811397 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.811403 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.811481 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.811655 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.811656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.811658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:34:46.811713 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.811715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.812375 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.812376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.813217 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.813219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.813778 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.813779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.813843 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.813847 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.814189 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.814238 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":53,"threads_started":1}
I20260927 18:34:46.814361 13479 leader_election.cc:302] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bbfe20b5ab46a68801b3f800f4b6cf, f454bb5d7ff940a09cd681047982a3b0; no voters:
I20260927 18:34:46.814505 13763 raft_consensus.cc:2781] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:46.814545 13763 raft_consensus.cc:482] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:46.814610 13763 raft_consensus.cc:3037] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.814572 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8471e726a742a889c6ba66c0d2c2e6" candidate_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8d406a6eca426ab409fc982b3e38ff" is_pre_election: true
I20260927 18:34:46.814648 13406 raft_consensus.cc:2454] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18bbfe20b5ab46a68801b3f800f4b6cf in term 0.
I20260927 18:34:46.815274 13763 raft_consensus.cc:504] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.815409 13763 leader_election.cc:288] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [CANDIDATE]: Term 1 election: Requested vote from peers fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305), f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437)
I20260927 18:34:46.815532 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8471e726a742a889c6ba66c0d2c2e6" candidate_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8d406a6eca426ab409fc982b3e38ff"
I20260927 18:34:46.815626 13406 raft_consensus.cc:3037] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.815600 13686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8471e726a742a889c6ba66c0d2c2e6" candidate_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f454bb5d7ff940a09cd681047982a3b0"
I20260927 18:34:46.815660 13686 raft_consensus.cc:3037] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:46.816349 13686 raft_consensus.cc:2454] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18bbfe20b5ab46a68801b3f800f4b6cf in term 1.
I20260927 18:34:46.816366 13406 raft_consensus.cc:2454] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18bbfe20b5ab46a68801b3f800f4b6cf in term 1.
I20260927 18:34:46.816511 13479 leader_election.cc:302] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bbfe20b5ab46a68801b3f800f4b6cf, f454bb5d7ff940a09cd681047982a3b0; no voters:
I20260927 18:34:46.816615 13763 raft_consensus.cc:2781] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:46.816735 13763 raft_consensus.cc:686] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 LEADER]: Becoming Leader. State: Replica: 18bbfe20b5ab46a68801b3f800f4b6cf, State: Running, Role: LEADER
I20260927 18:34:46.816817 13763 consensus_queue.cc:237] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.817150 13799 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: TxnStatusTablet state changed. Reason: New leader 18bbfe20b5ab46a68801b3f800f4b6cf. Latest consensus state: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.817251 13799 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:46.817408 13801 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:34:46.817499 13801 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:34:46.817481 13247 catalog_manager.cc:5495] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194). New cstate: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20260927 18:34:46.824297 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.821758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:34043 (local address 127.0.0.1:48284)
0927 18:34:46.821831 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.821842 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.821886 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.822070 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.822071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.822072 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:46.822118 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.822120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.822959 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.822960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.823457 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.823459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.823889 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.823890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.823930 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.823942 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.824179 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.824184 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.824218 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.824249 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 18:34:46.824311 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.821858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.5.151.194:34043)
0927 18:34:46.821915 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.821917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.821925 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:46.821955 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.821956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.821974 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.822052 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:46.822179 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.822896 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.823546 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.823816 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.823878 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.824019 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.824120 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.824189 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.824224 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.824243 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:34:46.825521 13686 raft_consensus.cc:1261] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Refusing update from remote peer 18bbfe20b5ab46a68801b3f800f4b6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:46.825565 13406 raft_consensus.cc:1261] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Refusing update from remote peer 18bbfe20b5ab46a68801b3f800f4b6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:46.825759 13799 consensus_queue.cc:1035] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:46.825870 13763 consensus_queue.cc:1035] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:46.826337 13765 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: TxnStatusTablet state changed. Reason: New leader 18bbfe20b5ab46a68801b3f800f4b6cf. Latest consensus state: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.826421 13765 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:46.826438 13767 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: TxnStatusTablet state changed. Reason: New leader 18bbfe20b5ab46a68801b3f800f4b6cf. Latest consensus state: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.826517 13767 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:46.827415 13799 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.827489 13799 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:46.827533 13799 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.827574 13799 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P 18bbfe20b5ab46a68801b3f800f4b6cf: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:34:46.827836 13767 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.827917 13767 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P fd8d406a6eca426ab409fc982b3e38ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:46.828146 13765 tablet_replica.cc:439] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } } }
I20260927 18:34:46.828218 13765 tablet_replica.cc:442] T 9e8471e726a742a889c6ba66c0d2c2e6 P f454bb5d7ff940a09cd681047982a3b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:34:46.828362 13770 mvcc.cc:205] Tried to move back new op lower bound from 7334027619635363840 to 7334027619602149376. Current Snapshot: MvccSnapshot[applied={T|T < 7334027619635363840 or (T in {7334027619635363840})}]
I20260927 18:34:46.831535 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.829152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.5.151.195:35437)
0927 18:34:46.829185 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.829187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.829195 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:46.829260 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.829260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.829275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.829330 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:46.829432 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.829849 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.830886 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.831056 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.831109 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.831260 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.831341 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.831386 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.831466 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.831480 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:34:46.836575 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.833719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:34043 (local address 127.5.151.195:54247)
0927 18:34:46.833873 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.833881 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.833926 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.834031 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.834031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.834032 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:46.834073 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.834074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.834889 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.834890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.835837 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.835839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.836248 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.836249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.836292 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.836303 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.836451 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.836453 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.836495 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.836524 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260927 18:34:46.836582 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.833772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:54247 (local address 127.5.151.194:34043)
0927 18:34:46.833809 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.833810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.833922 (+ 112us) server_negotiation.cc:407] Connection header received
0927 18:34:46.833963 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.833964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.833973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.834030 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:46.834106 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.834826 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.835912 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.836181 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.836244 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.836372 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.836401 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.836447 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.836497 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.836514 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:34:46.838737 13406 raft_consensus.cc:1261] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Refusing update from remote peer f454bb5d7ff940a09cd681047982a3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:46.838747 13546 raft_consensus.cc:1261] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Refusing update from remote peer f454bb5d7ff940a09cd681047982a3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:46.838989 13765 consensus_queue.cc:1035] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:46.839121 13786 consensus_queue.cc:1035] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:46.847155 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.844754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:47111 (local address 127.5.151.195:35437)
0927 18:34:46.844789 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.844792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.844904 (+ 112us) server_negotiation.cc:407] Connection header received
0927 18:34:46.844962 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.844963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.844982 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.845028 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:46.845161 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.845595 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.846351 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.846650 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.846723 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.846940 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.846975 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.847032 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.847070 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.847090 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:34:46.847199 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.844608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35437 (local address 127.5.151.194:47111)
0927 18:34:46.844834 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:46.844844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:46.844915 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:46.845030 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:46.845031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:46.845032 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:34:46.845091 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.845092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.845684 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.845685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.846269 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:46.846271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.846762 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.846763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:46.846825 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.846840 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:34:46.847032 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:34:46.847033 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 18:34:46.847068 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 18:34:46.847122 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":67,"threads_started":1}
I20260927 18:34:46.856386 13247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:46.856504 13247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:46.857113 13247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Sending DeleteTablet for 3 replicas of tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c
I20260927 18:34:46.857338 13526 tablet_service.cc:1514] Processing DeleteTablet for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:46 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:48280
I20260927 18:34:46.857362 13666 tablet_service.cc:1514] Processing DeleteTablet for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:46 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:56108
I20260927 18:34:46.857426 13781 tablet_replica.cc:330] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: stopping tablet replica
I20260927 18:34:46.857436 13780 tablet_replica.cc:330] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: stopping tablet replica
I20260927 18:34:46.857477 13780 raft_consensus.cc:2229] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:46.857481 13781 raft_consensus.cc:2229] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:46.857517 13781 raft_consensus.cc:2258] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:46.857548 13780 raft_consensus.cc:2258] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:46.857745 13386 tablet_service.cc:1514] Processing DeleteTablet for tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:46 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50666
I20260927 18:34:46.857821 13779 tablet_replica.cc:330] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: stopping tablet replica
I20260927 18:34:46.857867 13779 raft_consensus.cc:2229] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:46.857897 13779 raft_consensus.cc:2258] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:46.857903 13781 ts_tablet_manager.cc:1905] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:46.859563 13781 ts_tablet_manager.cc:1918] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:46.859614 13781 log.cc:1196] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/1592d4cb8a8f4409baaa4f05a1eb3f6c
I20260927 18:34:46.859829 13781 ts_tablet_manager.cc:1939] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting consensus metadata
I20260927 18:34:46.860196 13235 catalog_manager.cc:4841] TS 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043): tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c (table test-table [id=3e12cca25a3745ebbe7eadf59ae37e3b]) successfully deleted
I20260927 18:34:46.860488 13780 ts_tablet_manager.cc:1905] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:46.861968 13780 ts_tablet_manager.cc:1918] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:46.862017 13780 log.cc:1196] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/1592d4cb8a8f4409baaa4f05a1eb3f6c
I20260927 18:34:46.862160 13779 ts_tablet_manager.cc:1905] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:46.862226 13780 ts_tablet_manager.cc:1939] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P f454bb5d7ff940a09cd681047982a3b0: Deleting consensus metadata
I20260927 18:34:46.862984 13235 catalog_manager.cc:4841] TS f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437): tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c (table test-table [id=3e12cca25a3745ebbe7eadf59ae37e3b]) successfully deleted
I20260927 18:34:46.863649 13779 ts_tablet_manager.cc:1918] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:46.863710 13779 log.cc:1196] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/1592d4cb8a8f4409baaa4f05a1eb3f6c
I20260927 18:34:46.863916 13779 ts_tablet_manager.cc:1939] T 1592d4cb8a8f4409baaa4f05a1eb3f6c P fd8d406a6eca426ab409fc982b3e38ff: Deleting consensus metadata
I20260927 18:34:46.864204 13234 catalog_manager.cc:4841] TS fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305): tablet 1592d4cb8a8f4409baaa4f05a1eb3f6c (table test-table [id=3e12cca25a3745ebbe7eadf59ae37e3b]) successfully deleted
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:34:46.894516 13298 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:46.887561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.5.151.254:43127)
0927 18:34:46.887616 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.887618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.887770 (+ 152us) server_negotiation.cc:407] Connection header received
0927 18:34:46.887835 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.887836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.887848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.887923 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.888661 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.889495 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.890631 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.890947 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.891022 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.894425 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41574: BlockingRecv error: recv got EOF from 127.0.0.1:41574 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 18:34:46 dist-test-slave-19nl krb5kdc[13204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM
I20260927 18:34:46.904114 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:46.895239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.5.151.254:43127)
0927 18:34:46.895276 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:46.895277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:46.895693 (+ 416us) server_negotiation.cc:407] Connection header received
0927 18:34:46.895760 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:46.895761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:46.895771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:46.895831 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:46.896374 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.897126 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.898165 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:46.898479 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:46.898600 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:46.901194 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:46.901196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:46.901196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:46.901204 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:46.902874 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.903266 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.903267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.903268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.903331 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:46.903556 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:46.903556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:46.903557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:46.903701 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:46.903807 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:46.903946 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:34:46.904026 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:46.906122 13247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:46.906306 13247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:46.907725 13386 tablet_service.cc:1467] Processing CreateTablet for tablet 71a8b9944f9740baa060e6f030a262d8 (DEFAULT_TABLE table=test-table [id=09bf8879697e4057b0846cdffe704f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.907780 13666 tablet_service.cc:1467] Processing CreateTablet for tablet 71a8b9944f9740baa060e6f030a262d8 (DEFAULT_TABLE table=test-table [id=09bf8879697e4057b0846cdffe704f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.907860 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a8b9944f9740baa060e6f030a262d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.907912 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a8b9944f9740baa060e6f030a262d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.909155 13758 tablet_bootstrap.cc:492] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: Bootstrap starting.
I20260927 18:34:46.909283 13761 tablet_bootstrap.cc:492] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: Bootstrap starting.
I20260927 18:34:46.909466 13758 tablet_bootstrap.cc:654] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.909689 13761 tablet_bootstrap.cc:654] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.910071 13526 tablet_service.cc:1467] Processing CreateTablet for tablet 71a8b9944f9740baa060e6f030a262d8 (DEFAULT_TABLE table=test-table [id=09bf8879697e4057b0846cdffe704f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:46.910192 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a8b9944f9740baa060e6f030a262d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:46.910429 13758 tablet_bootstrap.cc:492] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: No bootstrap required, opened a new log
I20260927 18:34:46.910485 13758 ts_tablet_manager.cc:1397] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:46.910635 13758 raft_consensus.cc:348] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.910719 13758 raft_consensus.cc:374] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.910740 13758 raft_consensus.cc:729] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8d406a6eca426ab409fc982b3e38ff, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.910777 13758 consensus_queue.cc:260] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.910858 13758 ts_tablet_manager.cc:1428] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:46.911523 13757 tablet_bootstrap.cc:492] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: Bootstrap starting.
I20260927 18:34:46.911815 13757 tablet_bootstrap.cc:654] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:46.912472 13757 tablet_bootstrap.cc:492] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: No bootstrap required, opened a new log
I20260927 18:34:46.912524 13757 ts_tablet_manager.cc:1397] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:46.912679 13757 raft_consensus.cc:348] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.912766 13757 raft_consensus.cc:374] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.912788 13757 raft_consensus.cc:729] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bbfe20b5ab46a68801b3f800f4b6cf, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.912832 13757 consensus_queue.cc:260] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.912910 13757 ts_tablet_manager.cc:1428] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:46.913733 13761 tablet_bootstrap.cc:492] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: No bootstrap required, opened a new log
I20260927 18:34:46.913784 13761 ts_tablet_manager.cc:1397] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 18:34:46.913908 13761 raft_consensus.cc:348] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.913969 13761 raft_consensus.cc:374] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:46.913988 13761 raft_consensus.cc:729] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f454bb5d7ff940a09cd681047982a3b0, State: Initialized, Role: FOLLOWER
I20260927 18:34:46.914027 13761 consensus_queue.cc:260] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:46.914124 13761 ts_tablet_manager.cc:1428] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:47.036927 13767 raft_consensus.cc:482] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:47.037052 13767 raft_consensus.cc:504] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.037231 13767 leader_election.cc:288] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043), f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437)
I20260927 18:34:47.037442 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a8b9944f9740baa060e6f030a262d8" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" is_pre_election: true
I20260927 18:34:47.037472 13686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a8b9944f9740baa060e6f030a262d8" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f454bb5d7ff940a09cd681047982a3b0" is_pre_election: true
I20260927 18:34:47.037554 13686 raft_consensus.cc:2454] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 0.
I20260927 18:34:47.037554 13546 raft_consensus.cc:2454] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 0.
I20260927 18:34:47.037715 13339 leader_election.cc:302] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bbfe20b5ab46a68801b3f800f4b6cf, fd8d406a6eca426ab409fc982b3e38ff; no voters:
I20260927 18:34:47.037825 13767 raft_consensus.cc:2781] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:47.037870 13767 raft_consensus.cc:482] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:47.037896 13767 raft_consensus.cc:3037] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:47.038682 13767 raft_consensus.cc:504] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.038815 13767 leader_election.cc:288] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 election: Requested vote from peers 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043), f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437)
I20260927 18:34:47.038968 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a8b9944f9740baa060e6f030a262d8" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf"
I20260927 18:34:47.038973 13686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a8b9944f9740baa060e6f030a262d8" candidate_uuid: "fd8d406a6eca426ab409fc982b3e38ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f454bb5d7ff940a09cd681047982a3b0"
I20260927 18:34:47.039038 13686 raft_consensus.cc:3037] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:47.039038 13546 raft_consensus.cc:3037] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:47.039763 13546 raft_consensus.cc:2454] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 1.
I20260927 18:34:47.039763 13686 raft_consensus.cc:2454] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8d406a6eca426ab409fc982b3e38ff in term 1.
I20260927 18:34:47.039954 13339 leader_election.cc:302] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f454bb5d7ff940a09cd681047982a3b0, fd8d406a6eca426ab409fc982b3e38ff; no voters:
I20260927 18:34:47.040042 13767 raft_consensus.cc:2781] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:47.040156 13767 raft_consensus.cc:686] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 1 LEADER]: Becoming Leader. State: Replica: fd8d406a6eca426ab409fc982b3e38ff, State: Running, Role: LEADER
I20260927 18:34:47.040197 13767 consensus_queue.cc:237] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.040766 13248 catalog_manager.cc:5495] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193). New cstate: current_term: 1 leader_uuid: "fd8d406a6eca426ab409fc982b3e38ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20260927 18:34:47.060400 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:47.058040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.5.151.193:37305)
0927 18:34:47.058108 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:47.058110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:47.058119 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:47.058178 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:47.058180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:47.058197 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:47.058257 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:47.058381 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:47.058933 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:47.059697 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:47.059891 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:47.059963 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:47.060123 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:47.060165 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:47.060225 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:47.060287 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 18:34:47.060307 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:34:47.061159 13686 raft_consensus.cc:1261] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Refusing update from remote peer fd8d406a6eca426ab409fc982b3e38ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:47.061192 13546 raft_consensus.cc:1261] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Refusing update from remote peer fd8d406a6eca426ab409fc982b3e38ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:47.061389 13832 consensus_queue.cc:1035] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:47.061486 13767 consensus_queue.cc:1035] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:47.064307 13248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:47.064394 13248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:47.064994 13248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Sending DeleteTablet for 3 replicas of tablet 71a8b9944f9740baa060e6f030a262d8
I20260927 18:34:47.065243 13386 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a8b9944f9740baa060e6f030a262d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:47 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50666
I20260927 18:34:47.065253 13666 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a8b9944f9740baa060e6f030a262d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:47 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:56108
I20260927 18:34:47.065332 13779 tablet_replica.cc:330] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: stopping tablet replica
I20260927 18:34:47.065351 13526 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a8b9944f9740baa060e6f030a262d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:47 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:48280
I20260927 18:34:47.065389 13779 raft_consensus.cc:2229] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:47.065430 13781 tablet_replica.cc:330] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: stopping tablet replica
I20260927 18:34:47.065436 13780 tablet_replica.cc:330] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: stopping tablet replica
I20260927 18:34:47.065459 13779 raft_consensus.cc:2258] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:47.065485 13781 raft_consensus.cc:2229] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:47.065503 13780 raft_consensus.cc:2229] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:47.065532 13781 raft_consensus.cc:2258] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:47.065546 13780 raft_consensus.cc:2258] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:47.065621 13248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:34:47.065762 13248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:34:47.065862 13779 ts_tablet_manager.cc:1905] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:47.065925 13781 ts_tablet_manager.cc:1905] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:47.065932 13780 ts_tablet_manager.cc:1905] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:47.067173 13781 ts_tablet_manager.cc:1918] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:47.067211 13781 log.cc:1196] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/71a8b9944f9740baa060e6f030a262d8
I20260927 18:34:47.067337 13779 ts_tablet_manager.cc:1918] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:47.067374 13780 ts_tablet_manager.cc:1918] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:34:47.067387 13779 log.cc:1196] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/71a8b9944f9740baa060e6f030a262d8
I20260927 18:34:47.067425 13780 log.cc:1196] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/71a8b9944f9740baa060e6f030a262d8
I20260927 18:34:47.067430 13781 ts_tablet_manager.cc:1939] T 71a8b9944f9740baa060e6f030a262d8 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting consensus metadata
I20260927 18:34:47.067631 13779 ts_tablet_manager.cc:1939] T 71a8b9944f9740baa060e6f030a262d8 P fd8d406a6eca426ab409fc982b3e38ff: Deleting consensus metadata
I20260927 18:34:47.067648 13780 ts_tablet_manager.cc:1939] T 71a8b9944f9740baa060e6f030a262d8 P f454bb5d7ff940a09cd681047982a3b0: Deleting consensus metadata
I20260927 18:34:47.067793 13235 catalog_manager.cc:4841] TS 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043): tablet 71a8b9944f9740baa060e6f030a262d8 (table test-table [id=09bf8879697e4057b0846cdffe704f88]) successfully deleted
I20260927 18:34:47.068037 13234 catalog_manager.cc:4841] TS fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305): tablet 71a8b9944f9740baa060e6f030a262d8 (table test-table [id=09bf8879697e4057b0846cdffe704f88]) successfully deleted
I20260927 18:34:47.068099 13235 catalog_manager.cc:4841] TS f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437): tablet 71a8b9944f9740baa060e6f030a262d8 (table test-table [id=09bf8879697e4057b0846cdffe704f88]) successfully deleted
I20260927 18:34:47.068687 13386 tablet_service.cc:1467] Processing CreateTablet for tablet b2cda14faf5545f7bf6e99f52fab5bf2 (DEFAULT_TABLE table=test-table [id=8fb0948627c24a5cb439dce43451e4f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:47.068799 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2cda14faf5545f7bf6e99f52fab5bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:47.068821 13526 tablet_service.cc:1467] Processing CreateTablet for tablet b2cda14faf5545f7bf6e99f52fab5bf2 (DEFAULT_TABLE table=test-table [id=8fb0948627c24a5cb439dce43451e4f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:47.068837 13666 tablet_service.cc:1467] Processing CreateTablet for tablet b2cda14faf5545f7bf6e99f52fab5bf2 (DEFAULT_TABLE table=test-table [id=8fb0948627c24a5cb439dce43451e4f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:34:47.068922 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2cda14faf5545f7bf6e99f52fab5bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:47.068925 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2cda14faf5545f7bf6e99f52fab5bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:47.069868 13757 tablet_bootstrap.cc:492] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: Bootstrap starting.
I20260927 18:34:47.070003 13758 tablet_bootstrap.cc:492] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff: Bootstrap starting.
I20260927 18:34:47.070145 13757 tablet_bootstrap.cc:654] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:47.070245 13758 tablet_bootstrap.cc:654] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:47.070501 13761 tablet_bootstrap.cc:492] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: Bootstrap starting.
I20260927 18:34:47.070676 13757 tablet_bootstrap.cc:492] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: No bootstrap required, opened a new log
I20260927 18:34:47.070727 13757 ts_tablet_manager.cc:1397] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:34:47.070822 13758 tablet_bootstrap.cc:492] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff: No bootstrap required, opened a new log
I20260927 18:34:47.070847 13761 tablet_bootstrap.cc:654] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:47.070873 13758 ts_tablet_manager.cc:1397] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:34:47.070855 13757 raft_consensus.cc:348] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.070948 13757 raft_consensus.cc:374] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:47.070976 13757 raft_consensus.cc:729] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bbfe20b5ab46a68801b3f800f4b6cf, State: Initialized, Role: FOLLOWER
I20260927 18:34:47.070978 13758 raft_consensus.cc:348] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.071023 13757 consensus_queue.cc:260] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.071074 13758 raft_consensus.cc:374] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:47.071108 13758 raft_consensus.cc:729] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8d406a6eca426ab409fc982b3e38ff, State: Initialized, Role: FOLLOWER
I20260927 18:34:47.071115 13757 ts_tablet_manager.cc:1428] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:47.071151 13758 consensus_queue.cc:260] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.071256 13758 ts_tablet_manager.cc:1428] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:47.071580 13761 tablet_bootstrap.cc:492] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: No bootstrap required, opened a new log
I20260927 18:34:47.071635 13761 ts_tablet_manager.cc:1397] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:47.071748 13761 raft_consensus.cc:348] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.071826 13761 raft_consensus.cc:374] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:47.071849 13761 raft_consensus.cc:729] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f454bb5d7ff940a09cd681047982a3b0, State: Initialized, Role: FOLLOWER
I20260927 18:34:47.071894 13761 consensus_queue.cc:260] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.071985 13761 ts_tablet_manager.cc:1428] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:47.269528 13814 raft_consensus.cc:482] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:34:47.269644 13814 raft_consensus.cc:504] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.269883 13814 leader_election.cc:288] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305), 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043)
I20260927 18:34:47.270190 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2cda14faf5545f7bf6e99f52fab5bf2" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" is_pre_election: true
I20260927 18:34:47.270315 13546 raft_consensus.cc:2454] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 0.
I20260927 18:34:47.270323 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2cda14faf5545f7bf6e99f52fab5bf2" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8d406a6eca426ab409fc982b3e38ff" is_pre_election: true
I20260927 18:34:47.270416 13406 raft_consensus.cc:2454] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 0.
I20260927 18:34:47.270459 13619 leader_election.cc:302] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bbfe20b5ab46a68801b3f800f4b6cf, f454bb5d7ff940a09cd681047982a3b0; no voters:
I20260927 18:34:47.270617 13814 raft_consensus.cc:2781] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:34:47.270668 13814 raft_consensus.cc:482] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:34:47.270705 13814 raft_consensus.cc:3037] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:47.271610 13814 raft_consensus.cc:504] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.271771 13814 leader_election.cc:288] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 election: Requested vote from peers fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305), 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043)
I20260927 18:34:47.271932 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2cda14faf5545f7bf6e99f52fab5bf2" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf"
I20260927 18:34:47.271934 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2cda14faf5545f7bf6e99f52fab5bf2" candidate_uuid: "f454bb5d7ff940a09cd681047982a3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8d406a6eca426ab409fc982b3e38ff"
I20260927 18:34:47.272017 13546 raft_consensus.cc:3037] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:47.272017 13406 raft_consensus.cc:3037] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:47.272789 13546 raft_consensus.cc:2454] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 1.
I20260927 18:34:47.272789 13406 raft_consensus.cc:2454] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f454bb5d7ff940a09cd681047982a3b0 in term 1.
I20260927 18:34:47.272977 13619 leader_election.cc:302] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bbfe20b5ab46a68801b3f800f4b6cf, f454bb5d7ff940a09cd681047982a3b0; no voters:
I20260927 18:34:47.273074 13814 raft_consensus.cc:2781] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:47.273146 13814 raft_consensus.cc:686] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 1 LEADER]: Becoming Leader. State: Replica: f454bb5d7ff940a09cd681047982a3b0, State: Running, Role: LEADER
I20260927 18:34:47.273200 13814 consensus_queue.cc:237] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } }
I20260927 18:34:47.273819 13248 catalog_manager.cc:5495] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "f454bb5d7ff940a09cd681047982a3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f454bb5d7ff940a09cd681047982a3b0" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35437 } health_report { overall_health: HEALTHY } } }
I20260927 18:34:47.301033 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:47.298998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.5.151.195:35437)
0927 18:34:47.299044 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:47.299046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:47.299055 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:47.299108 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:47.299109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:47.299116 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:47.299187 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:34:47.299277 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:47.299761 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:47.300514 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:47.300685 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:47.300733 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:47.300833 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:34:47.300856 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:34:47.300888 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:47.300918 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 18:34:47.300933 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 18:34:47.303027 13546 raft_consensus.cc:1261] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Refusing update from remote peer f454bb5d7ff940a09cd681047982a3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:47.303030 13406 raft_consensus.cc:1261] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Refusing update from remote peer f454bb5d7ff940a09cd681047982a3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:34:47.303253 13765 consensus_queue.cc:1035] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18bbfe20b5ab46a68801b3f800f4b6cf" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:47.303352 13814 consensus_queue.cc:1035] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8d406a6eca426ab409fc982b3e38ff" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 37305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:34:47.313043 13247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:47.313144 13247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:34:47.313690 13247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241fb61a681048fbac8e08ea0ae02309: Sending DeleteTablet for 3 replicas of tablet b2cda14faf5545f7bf6e99f52fab5bf2
I20260927 18:34:47.313906 13526 tablet_service.cc:1514] Processing DeleteTablet for tablet b2cda14faf5545f7bf6e99f52fab5bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:47 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:48280
I20260927 18:34:47.313911 13386 tablet_service.cc:1514] Processing DeleteTablet for tablet b2cda14faf5545f7bf6e99f52fab5bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:47 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50666
I20260927 18:34:47.314002 13781 tablet_replica.cc:330] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: stopping tablet replica
I20260927 18:34:47.314023 13666 tablet_service.cc:1514] Processing DeleteTablet for tablet b2cda14faf5545f7bf6e99f52fab5bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:34:47 UTC) from {username='oryx', principal='oryx/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:56108
I20260927 18:34:47.314055 13781 raft_consensus.cc:2229] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:47.314095 13781 raft_consensus.cc:2258] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:47.314121 13779 tablet_replica.cc:330] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff: stopping tablet replica
I20260927 18:34:47.314178 13779 raft_consensus.cc:2229] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:34:47.314219 13779 raft_consensus.cc:2258] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:47.314306 13780 tablet_replica.cc:330] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: stopping tablet replica
I20260927 18:34:47.314365 13780 raft_consensus.cc:2229] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:34:47.314436 13780 raft_consensus.cc:2258] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:34:47.314484 13781 ts_tablet_manager.cc:1905] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:47.314587 13779 ts_tablet_manager.cc:1905] T b2cda14faf5545f7bf6e99f52fab5bf2 P fd8d406a6eca426ab409fc982b3e38ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:47.314791 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 13321
I20260927 18:34:47.316260 13781 ts_tablet_manager.cc:1918] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:47.316324 13781 log.cc:1196] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/b2cda14faf5545f7bf6e99f52fab5bf2
I20260927 18:34:47.316533 13781 ts_tablet_manager.cc:1939] T b2cda14faf5545f7bf6e99f52fab5bf2 P 18bbfe20b5ab46a68801b3f800f4b6cf: Deleting consensus metadata
I20260927 18:34:47.316915 13235 catalog_manager.cc:4841] TS 18bbfe20b5ab46a68801b3f800f4b6cf (127.5.151.194:34043): tablet b2cda14faf5545f7bf6e99f52fab5bf2 (table test-table [id=8fb0948627c24a5cb439dce43451e4f4]) successfully deleted
I20260927 18:34:47.317224 13780 ts_tablet_manager.cc:1905] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:34:47.318341 13780 ts_tablet_manager.cc:1918] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:34:47.318418 13780 log.cc:1196] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/b2cda14faf5545f7bf6e99f52fab5bf2
I20260927 18:34:47.318682 13780 ts_tablet_manager.cc:1939] T b2cda14faf5545f7bf6e99f52fab5bf2 P f454bb5d7ff940a09cd681047982a3b0: Deleting consensus metadata
I20260927 18:34:47.319010 13235 catalog_manager.cc:4841] TS f454bb5d7ff940a09cd681047982a3b0 (127.5.151.195:35437): tablet b2cda14faf5545f7bf6e99f52fab5bf2 (table test-table [id=8fb0948627c24a5cb439dce43451e4f4]) successfully deleted
W20260927 18:34:47.321800 13234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:37305 (error 108)
I20260927 18:34:47.322091 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 13461
I20260927 18:34:47.326812 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:47.322158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:37305 (local address 127.0.0.1:50690)
0927 18:34:47.326695 (+ 4537us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:47.326752 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:37305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4521,"thread_start_us":38,"threads_started":1}
W20260927 18:34:47.326927 13234 catalog_manager.cc:4584] TS fd8d406a6eca426ab409fc982b3e38ff (127.5.151.193:37305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2cda14faf5545f7bf6e99f52fab5bf2: Network error: Client connection negotiation failed: client connection to 127.5.151.193:37305: connect: Connection refused (error 111)
I20260927 18:34:47.328356 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 13601
I20260927 18:34:47.333736 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 13217
2026-09-27T18:34:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2775 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:47 dist-test-slave-19nl krb5kdc[13847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:47 dist-test-slave-19nl krb5kdc[13847](info): set up 2 sockets
Sep 27 18:34:47 dist-test-slave-19nl krb5kdc[13847](info): commencing operation
krb5kdc: starting...
W20260927 18:34:48.334988 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:48 dist-test-slave-19nl krb5kdc[13847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:48.443468 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34007
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:34007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:48.521107 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:48.521360 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:48.521399 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:48.523028 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:48.523129 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:48.523159 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:48.523188 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:48.523218 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:48.524911 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:34007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:48.525188 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:48.525413 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:48.530226 13865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:48.530197 13866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:48.530292 13860 server_base.cc:1034] running on GCE node
W20260927 18:34:48.530197 13868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:48.530669 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:48.530920 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:48.532068 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790534088532057 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T18:34:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1205 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:34:48 dist-test-slave-19nl krb5kdc[13873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:34:48 dist-test-slave-19nl krb5kdc[13873](info): set up 2 sockets
Sep 27 18:34:48 dist-test-slave-19nl krb5kdc[13873](info): commencing operation
krb5kdc: starting...
W20260927 18:34:49.586861 5727 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:34:49 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:34:49.688859 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45915
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:49.764173 13886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:49.764380 13886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:49.764413 13886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:49.765414 13886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:49.765473 13886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:49.765494 13886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:49.765513 13886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:49.765539 13886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:49.766968 13886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:49.767231 13886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:49.767464 13886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:49.772022 13892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:49.772033 13891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:49.772066 13894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:49.772199 13886 server_base.cc:1034] running on GCE node
I20260927 18:34:49.772480 13886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:49.772764 13886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:49.773929 13886 hybrid_clock.cc:648] HybridClock initialized: now 1790534089773917 us; error 22 us; skew 500 ppm
Sep 27 18:34:49 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534089, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:49.776662 13886 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:49.777119 13886 webserver.cc:460] Webserver started at http://127.5.151.254:33971/ using document root <none> and password file <none>
I20260927 18:34:49.777343 13886 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:49.777406 13886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:49.777534 13886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:49.778508 13886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "47cafef63d41440d8073eb07ac7cfd02"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "cca52de1b174479b6064532695aaddf6"
server_key_iv: "d34e4122cffd7e75449824a7eab1f5c4"
server_key_version: "encryptionkey@0"
I20260927 18:34:49.778916 13886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "47cafef63d41440d8073eb07ac7cfd02"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "cca52de1b174479b6064532695aaddf6"
server_key_iv: "d34e4122cffd7e75449824a7eab1f5c4"
server_key_version: "encryptionkey@0"
I20260927 18:34:49.780158 13886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:49.780784 13901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:49.780936 13886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:49.781033 13886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "47cafef63d41440d8073eb07ac7cfd02"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "cca52de1b174479b6064532695aaddf6"
server_key_iv: "d34e4122cffd7e75449824a7eab1f5c4"
server_key_version: "encryptionkey@0"
I20260927 18:34:49.781116 13886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:49.790795 13886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:49.792464 13886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:49.792613 13886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:49.796317 13886 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:45915
I20260927 18:34:49.796343 13953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:45915 every 8 connection(s)
I20260927 18:34:49.796679 13886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:49.797415 13954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:34:49.799461 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Bootstrap starting.
I20260927 18:34:49.799957 13954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Neither blocks nor log segments found. Creating new log.
I20260927 18:34:49.800187 13954 log.cc:824] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:49.800765 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: No bootstrap required, opened a new log
I20260927 18:34:49.802181 13954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:49.802299 13954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:34:49.802317 13954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cafef63d41440d8073eb07ac7cfd02, State: Initialized, Role: FOLLOWER
I20260927 18:34:49.802284 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 13886
I20260927 18:34:49.802399 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:34:49.802452 13954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:49.802543 13954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:49.802596 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:49.802634 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:34:49.802636 5727 external_mini_cluster.cc:1428] Setting key e68f07cb9b5e6db14a4e790cbf80f7dc
I20260927 18:34:49.803185 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:49.803290 13954 leader_election.cc:302] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cafef63d41440d8073eb07ac7cfd02; no voters:
I20260927 18:34:49.803452 13954 leader_election.cc:288] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:34:49.803481 13959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:34:49.803622 13959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 LEADER]: Becoming Leader. State: Replica: 47cafef63d41440d8073eb07ac7cfd02, State: Running, Role: LEADER
I20260927 18:34:49.803712 13954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:49.803711 13959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:49.804386 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47cafef63d41440d8073eb07ac7cfd02. Latest consensus state: current_term: 1 leader_uuid: "47cafef63d41440d8073eb07ac7cfd02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } } }
I20260927 18:34:49.804459 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:49.804589 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47cafef63d41440d8073eb07ac7cfd02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } } }
I20260927 18:34:49.804664 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:49.804777 13968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:49.805403 13968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:49.806733 13968 catalog_manager.cc:1347] Generated new cluster ID: 5819ebb354e0405fa724ecbaaf73a5bb
I20260927 18:34:49.806783 13968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:34:49 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:49.813369 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:49.803091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.5.151.254:45915)
0927 18:34:49.803257 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:49.803265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:49.803295 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:34:49.803415 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:49.803421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:49.803502 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:49.803645 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:49.804584 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:49.806200 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:49.807118 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:49.807671 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:49.807768 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:49.809964 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:49.809968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:49.809968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:49.809979 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:49.811749 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:49.812148 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:49.812151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:49.812151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:49.812255 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:49.812452 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:49.812453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:49.812453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:49.812696 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:49.812845 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:49.812964 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:34:49.813081 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260927 18:34:49.825152 13968 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:34:49.825603 13968 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:49.835050 13968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Generated new TSK 0
I20260927 18:34:49.835201 13968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:34:49.858843 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:45915
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:49.937546 13980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:49.937753 13980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:49.937775 13980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:49.938849 13980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:49.938882 13980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:49.938908 13980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:49.940217 13980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:49.940394 13980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:49.940553 13980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:49.945374 13985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:49.945497 13980 server_base.cc:1034] running on GCE node
W20260927 18:34:49.945384 13986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:49.945518 13988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:49.945807 13980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:49.946033 13980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:49.947186 13980 hybrid_clock.cc:648] HybridClock initialized: now 1790534089947164 us; error 33 us; skew 500 ppm
Sep 27 18:34:49 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534089, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:49.949609 13980 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:34:49.949996 13980 webserver.cc:460] Webserver started at http://127.5.151.193:34693/ using document root <none> and password file <none>
I20260927 18:34:49.950224 13980 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:49.950289 13980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:49.950431 13980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:49.951458 13980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "ff3095d8b4704ad293e4f4f02bb4e315"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "5bb8932ad34afa1ccf7bcb029ec765a6"
server_key_iv: "b3255f6938aed99aaf3867557d311f08"
server_key_version: "encryptionkey@0"
I20260927 18:34:49.951818 13980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "ff3095d8b4704ad293e4f4f02bb4e315"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "5bb8932ad34afa1ccf7bcb029ec765a6"
server_key_iv: "b3255f6938aed99aaf3867557d311f08"
server_key_version: "encryptionkey@0"
I20260927 18:34:49.953168 13980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:34:49.953827 13995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:49.953984 13980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:49.954077 13980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "ff3095d8b4704ad293e4f4f02bb4e315"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "5bb8932ad34afa1ccf7bcb029ec765a6"
server_key_iv: "b3255f6938aed99aaf3867557d311f08"
server_key_version: "encryptionkey@0"
I20260927 18:34:49.954151 13980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:49.973595 13980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:49.975735 13980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:49.975883 13980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:49.976086 13980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:49.976372 13980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:49.976423 13980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:49.976462 13980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:49.976488 13980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:49.982534 13980 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:34925
I20260927 18:34:49.982607 14108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:34925 every 8 connection(s)
I20260927 18:34:49.982970 13980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:34:49.985095 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 13980
I20260927 18:34:49.985220 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:34:49.985438 5727 external_mini_cluster.cc:1428] Setting key 7192b900f960d036e551e128b4ed4f8c
Sep 27 18:34:49 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534089, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:49.994005 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:49.983668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:43029)
0927 18:34:49.983828 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:49.983866 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:49.984409 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:49.984658 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:49.984662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:49.985416 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:49.985677 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:49.985693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:49.986609 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:49.986610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:49.988027 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:49.988029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:49.988583 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:49.988584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:49.988731 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:49.989269 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:49.989278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:49.990901 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:49.992745 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:49.992750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:49.992752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:49.992954 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:49.993158 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:49.993158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:49.993158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:49.993254 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:49.993545 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:49.993549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:49.993671 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 18:34:49.993753 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:34:49.993860 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260927 18:34:49.994009 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:49.983669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:43029 (local address 127.5.151.254:45915)
0927 18:34:49.983755 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:49.983758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:49.983935 (+ 177us) server_negotiation.cc:407] Connection header received
0927 18:34:49.984508 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:49.984509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:49.984521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:49.984577 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:49.985804 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:49.986495 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:49.988156 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:49.988479 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:49.988549 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:49.990988 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:49.990990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:49.990990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:49.990998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:49.992658 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:49.993025 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:49.993026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:49.993027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:49.993092 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:49.993318 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:49.993319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:49.993319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:49.993479 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:49.993554 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:49.993806 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 18:34:49.993896 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:34:49.994577 14109 heartbeater.cc:347] Connected to a master server at 127.5.151.254:45915
I20260927 18:34:49.994695 14109 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:49.994869 14109 heartbeater.cc:511] Master 127.5.151.254:45915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
I20260927 18:34:49.995384 13918 ts_manager.cc:194] Registered new tserver with Master: ff3095d8b4704ad293e4f4f02bb4e315 (127.5.151.193:34925)
I20260927 18:34:49.995950 13918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:43029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:34:50.012763 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:45915
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:50.088543 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:50.088738 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:50.088760 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:50.089790 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:50.089821 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:50.089846 14114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:50.091123 14114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:50.091281 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:50.091435 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:50.095988 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.095971 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.095971 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.095997 14114 server_base.cc:1034] running on GCE node
I20260927 18:34:50.096339 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:50.096544 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:50.097713 14114 hybrid_clock.cc:648] HybridClock initialized: now 1790534090097699 us; error 28 us; skew 500 ppm
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:50.100134 14114 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:34:50.100489 14114 webserver.cc:460] Webserver started at http://127.5.151.194:42509/ using document root <none> and password file <none>
I20260927 18:34:50.100657 14114 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:50.100701 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:50.100796 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:50.101784 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "541d41a91b9d4dcba16c72e7d2e45afd"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "a12b5164a6114568aafa8059aa22d5e7"
server_key_iv: "9028f88d3fa5267022ee19e0dcd75776"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.102067 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "541d41a91b9d4dcba16c72e7d2e45afd"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "a12b5164a6114568aafa8059aa22d5e7"
server_key_iv: "9028f88d3fa5267022ee19e0dcd75776"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.103324 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:50.104107 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.104281 14114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:50.104372 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "541d41a91b9d4dcba16c72e7d2e45afd"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "a12b5164a6114568aafa8059aa22d5e7"
server_key_iv: "9028f88d3fa5267022ee19e0dcd75776"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.104442 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:50.119819 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:50.121604 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:50.121766 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:50.121987 14114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:50.122288 14114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:50.122339 14114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.122380 14114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:50.122409 14114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.128129 14114 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:34321
I20260927 18:34:50.128198 14242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:34321 every 8 connection(s)
I20260927 18:34:50.128535 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:50.137034 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.129242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:52917 (local address 127.5.151.254:45915)
0927 18:34:50.129288 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.129290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.129503 (+ 213us) server_negotiation.cc:407] Connection header received
0927 18:34:50.129971 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.129972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.129986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.130037 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.130978 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.131486 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.132456 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.132636 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.132692 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.134748 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:50.134749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:50.134749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:50.134755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:50.135914 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:50.136251 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:50.136251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:50.136252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:50.136298 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:50.136490 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:50.136490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:50.136491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:50.136583 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:50.136629 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:50.136850 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 18:34:50.136918 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:34:50.137109 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.129243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:52917)
0927 18:34:50.129388 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.129422 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.129891 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.130088 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.130091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.130669 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.130882 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.130888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.131604 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.131605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.132378 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.132380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.132717 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.132718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.132814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.133242 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.133250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.134658 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:50.135980 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:50.135986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:50.135988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:50.136192 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:50.136352 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:50.136352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:50.136353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:50.136441 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:50.136635 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:50.136639 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:50.136750 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 18:34:50.136827 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:34:50.136935 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260927 18:34:50.137163 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 14114
I20260927 18:34:50.137269 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:34:50.137486 5727 external_mini_cluster.cc:1428] Setting key 8b017b4e8c3b6f4280d0aa738008ffcd
I20260927 18:34:50.137560 14243 heartbeater.cc:347] Connected to a master server at 127.5.151.254:45915
I20260927 18:34:50.137787 14243 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:50.137965 14243 heartbeater.cc:511] Master 127.5.151.254:45915 requested a full tablet report, sending...
I20260927 18:34:50.138355 13918 ts_manager.cc:194] Registered new tserver with Master: 541d41a91b9d4dcba16c72e7d2e45afd (127.5.151.194:34321)
I20260927 18:34:50.138758 13918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:52917
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:34:50.158617 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:45915
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:50.235649 14248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:50.235826 14248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:50.235848 14248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:50.236963 14248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:50.237001 14248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:50.237056 14248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:50.238425 14248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:50.238660 14248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:50.238871 14248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:34:50.243532 14248 server_base.cc:1034] running on GCE node
W20260927 18:34:50.243462 14256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.243474 14254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.243495 14253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.243917 14248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:50.244156 14248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:50.245316 14248 hybrid_clock.cc:648] HybridClock initialized: now 1790534090245301 us; error 25 us; skew 500 ppm
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:50.247634 14248 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:34:50.248025 14248 webserver.cc:460] Webserver started at http://127.5.151.195:35717/ using document root <none> and password file <none>
I20260927 18:34:50.248242 14248 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:50.248307 14248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:50.248452 14248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:34:50.249425 14248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "af715081a1f44031b95b4b7d75dd7cf1"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "ca69e1f8cb2d6884f39b14bbcedb38f6"
server_key_iv: "eef1004b6b5aa7ac2471e0594096d90c"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.249768 14248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "af715081a1f44031b95b4b7d75dd7cf1"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "ca69e1f8cb2d6884f39b14bbcedb38f6"
server_key_iv: "eef1004b6b5aa7ac2471e0594096d90c"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.251133 14248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:34:50.251801 14263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.251959 14248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:50.252049 14248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "af715081a1f44031b95b4b7d75dd7cf1"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "ca69e1f8cb2d6884f39b14bbcedb38f6"
server_key_iv: "eef1004b6b5aa7ac2471e0594096d90c"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.252125 14248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:50.266826 14248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:50.268458 14248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:50.268601 14248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:50.268795 14248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:50.269061 14248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:50.269111 14248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.269148 14248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:50.269176 14248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.274700 14248 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:36639
I20260927 18:34:50.274737 14376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:36639 every 8 connection(s)
I20260927 18:34:50.275043 14248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:34:50.282732 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 14248
I20260927 18:34:50.282855 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:34:50.283034 5727 external_mini_cluster.cc:1428] Setting key e043cbd2e10742aed9b13e91e4f112dc
I20260927 18:34:50.283450 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.275769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:50051 (local address 127.5.151.254:45915)
0927 18:34:50.275838 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.275841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.276018 (+ 177us) server_negotiation.cc:407] Connection header received
0927 18:34:50.276449 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.276450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.276460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.276509 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.277345 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.277823 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.278780 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.278949 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.278998 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.280874 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:50.280876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:50.280876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:50.280885 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:50.282036 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:50.282382 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:50.282383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:50.282383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:50.282428 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:50.282644 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:50.282645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:50.282645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:50.282801 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:50.282878 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:50.283242 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 18:34:50.283332 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:34:50.283596 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.275737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:50051)
0927 18:34:50.275902 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.275935 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.276370 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.276556 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.276559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.277089 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.277272 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.277276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.277896 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.277897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.278715 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.278716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.279014 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.279014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.279102 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.279487 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.279494 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.280784 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:34:50.282103 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:50.282108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:50.282110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:50.282320 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:50.282487 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:34:50.282488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:34:50.282488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:34:50.282596 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:34:50.282924 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:34:50.282929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:34:50.283050 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 18:34:50.283193 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 18:34:50.283361 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260927 18:34:50.284060 14377 heartbeater.cc:347] Connected to a master server at 127.5.151.254:45915
I20260927 18:34:50.284165 14377 heartbeater.cc:464] Registering TS with master...
I20260927 18:34:50.284323 14377 heartbeater.cc:511] Master 127.5.151.254:45915 requested a full tablet report, sending...
I20260927 18:34:50.284695 13918 ts_manager.cc:194] Registered new tserver with Master: af715081a1f44031b95b4b7d75dd7cf1 (127.5.151.195:36639)
I20260927 18:34:50.285053 13918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:50051
I20260927 18:34:50.294803 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab.
Entry for principal oryx/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab.
WARNING: no policy specified for oryx/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab.
Entry for principal oryx/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab.
WARNING: no policy specified for oryx/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab.
Entry for principal oryx/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab.
I20260927 18:34:50.357117 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 13980
I20260927 18:34:50.361805 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 14114
I20260927 18:34:50.365837 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 14248
I20260927 18:34:50.369580 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 13886
I20260927 18:34:50.374382 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45915
--webserver_interface=127.5.151.254
--webserver_port=33971
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:34:50.450865 14386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:50.451109 14386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:50.451155 14386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:50.452327 14386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:34:50.452397 14386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:50.452421 14386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:50.452448 14386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:34:50.452476 14386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:34:50.453893 14386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=33971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:50.454165 14386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:50.454384 14386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:50.459105 14394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.459110 14391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.459285 14386 server_base.cc:1034] running on GCE node
W20260927 18:34:50.459110 14392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.459568 14386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:50.459861 14386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:50.461020 14386 hybrid_clock.cc:648] HybridClock initialized: now 1790534090461014 us; error 33 us; skew 500 ppm
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:50.463302 14386 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:34:50.463735 14386 webserver.cc:460] Webserver started at http://127.5.151.254:33971/ using document root <none> and password file <none>
I20260927 18:34:50.463951 14386 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:50.464030 14386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:50.465301 14386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:50.465927 14401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.466126 14386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:34:50.466220 14386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "47cafef63d41440d8073eb07ac7cfd02"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "cca52de1b174479b6064532695aaddf6"
server_key_iv: "d34e4122cffd7e75449824a7eab1f5c4"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.466583 14386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:50.495843 14386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:50.497838 14386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:50.498014 14386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:50.502085 14386 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:45915
I20260927 18:34:50.502121 14453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:45915 every 8 connection(s)
I20260927 18:34:50.502620 14386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:34:50.504065 14454 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:34:50.504719 14454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Bootstrap starting.
I20260927 18:34:50.508971 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 14386
I20260927 18:34:50.509327 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:34925
--local_ip_for_outbound_sockets=127.5.151.193
--tserver_master_addrs=127.5.151.254:45915
--webserver_port=34693
--webserver_interface=127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:34:50.512719 14454 log.cc:824] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Log is configured to *not* fsync() on all Append() calls
I20260927 18:34:50.514369 14454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:34:50.514693 14454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Bootstrap complete.
I20260927 18:34:50.516918 14454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:50.517135 14454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cafef63d41440d8073eb07ac7cfd02, State: Initialized, Role: FOLLOWER
I20260927 18:34:50.517307 14454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:50.517377 14454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:34:50.517436 14454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:34:50.517483 14454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:34:50.518478 14454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:50.518607 14454 leader_election.cc:302] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cafef63d41440d8073eb07ac7cfd02; no voters:
I20260927 18:34:50.518815 14454 leader_election.cc:288] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:34:50.518850 14458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:34:50.518965 14458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [term 2 LEADER]: Becoming Leader. State: Replica: 47cafef63d41440d8073eb07ac7cfd02, State: Running, Role: LEADER
I20260927 18:34:50.519050 14458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } }
I20260927 18:34:50.519086 14454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:34:50.519310 14458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47cafef63d41440d8073eb07ac7cfd02. Latest consensus state: current_term: 2 leader_uuid: "47cafef63d41440d8073eb07ac7cfd02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } } }
I20260927 18:34:50.519387 14458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:50.519316 14459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47cafef63d41440d8073eb07ac7cfd02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cafef63d41440d8073eb07ac7cfd02" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45915 } } }
I20260927 18:34:50.519552 14459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02 [sys.catalog]: This master's current role is: LEADER
I20260927 18:34:50.519574 14462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:34:50.520092 14462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:34:50.520262 14462 catalog_manager.cc:1259] Loaded cluster ID: 5819ebb354e0405fa724ecbaaf73a5bb
I20260927 18:34:50.520308 14462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:34:50.521487 14462 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:34:50.521692 14462 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47cafef63d41440d8073eb07ac7cfd02: Loaded TSK: 0
I20260927 18:34:50.521901 14462 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:34:50.618386 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:50.618678 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:50.618721 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:50.620318 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:50.620375 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:50.620412 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:34:50.622399 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.193.keytab
--principal=oryx/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:34925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=34693
--tserver_master_addrs=127.5.151.254:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:50.622677 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:50.622933 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:50.627952 14480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.627976 14481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.628057 14456 server_base.cc:1034] running on GCE node
W20260927 18:34:50.628163 14483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.628371 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:50.628628 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:50.629807 14456 hybrid_clock.cc:648] HybridClock initialized: now 1790534090629786 us; error 35 us; skew 500 ppm
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:50.632608 14456 init.cc:377] Logged in from keytab as oryx/127.5.151.193@KRBTEST.COM (short username oryx)
I20260927 18:34:50.632992 14456 webserver.cc:460] Webserver started at http://127.5.151.193:34693/ using document root <none> and password file <none>
I20260927 18:34:50.633206 14456 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:50.633286 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:50.634678 14456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:50.635290 14490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.635449 14456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.635540 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "ff3095d8b4704ad293e4f4f02bb4e315"
format_stamp: "Formatted at 2026-09-27 18:34:49 on dist-test-slave-19nl"
server_key: "5bb8932ad34afa1ccf7bcb029ec765a6"
server_key_iv: "b3255f6938aed99aaf3867557d311f08"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.635842 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:50.646085 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:50.648041 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:50.648188 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:50.648396 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:50.648680 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:50.648711 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.648761 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:50.648787 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.654853 14456 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:34925
I20260927 18:34:50.654953 14603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:34925 every 8 connection(s)
I20260927 18:34:50.655248 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.664382 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 14456
I20260927 18:34:50.664745 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:34321
--local_ip_for_outbound_sockets=127.5.151.194
--tserver_master_addrs=127.5.151.254:45915
--webserver_port=42509
--webserver_interface=127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.666309 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.655993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:59159)
0927 18:34:50.656158 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.656197 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.656639 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.659093 (+ 2454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.659097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.659699 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.659895 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.659900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.661043 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.661044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.661860 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.661863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.662511 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.662512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.662682 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.663268 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.663277 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.666046 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
W20260927 18:34:50.666424 14607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.656075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:59159 (local address 127.5.151.254:45915)
0927 18:34:50.658741 (+ 2666us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.658748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.658776 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:34:50.658874 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.658878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.658917 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.659058 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.660019 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.660924 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.661955 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.662401 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.662541 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.666240 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:59159: BlockingRecv error: recv got EOF from 127.5.151.193:59159 (error 108)
Metrics: {"server-negotiator.queue_time_us":2616,"thread_start_us":80,"threads_started":1}
W20260927 18:34:50.666913 14604 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:45915 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.151.254:45915: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.672878 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.667211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:55599)
0927 18:34:50.667343 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.667354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.667403 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.667561 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.667562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.667726 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.667786 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.667788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.668739 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.668740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.669538 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.669541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.670127 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.670128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.670204 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.670772 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.670777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.672813 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260927 18:34:50.673143 14607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.667295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:55599 (local address 127.5.151.254:45915)
0927 18:34:50.667354 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.667356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.667390 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:34:50.667457 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.667458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.667470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.667539 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.667873 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.668641 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.669637 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.670007 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.670128 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.673081 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:55599: BlockingRecv error: recv got EOF from 127.5.151.193:55599 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.679949 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.673143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:51275)
0927 18:34:50.674688 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.674701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.674761 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.674992 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.674994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.675151 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.675189 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.675191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.676141 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.676142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.676963 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.676965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.677519 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.677520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.677574 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.678024 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.678029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.679878 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1462}
W20260927 18:34:50.680080 14609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.673181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:51275 (local address 127.5.151.254:45915)
0927 18:34:50.673354 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.673356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.674760 (+ 1404us) server_negotiation.cc:407] Connection header received
0927 18:34:50.674828 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.674834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.674848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.674925 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.675278 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.676049 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.677056 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.677439 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.677510 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.679995 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:51275: BlockingRecv error: recv got EOF from 127.5.151.193:51275 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":108,"threads_started":1}
W20260927 18:34:50.680220 14604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:34:50.748047 14608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:50.748265 14608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:50.748301 14608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:50.749429 14608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:50.749491 14608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:50.749537 14608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:34:50.751014 14608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.194.keytab
--principal=oryx/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:34321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=42509
--tserver_master_addrs=127.5.151.254:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:50.751258 14608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:50.751466 14608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:50.756258 14615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.756281 14617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.756309 14608 server_base.cc:1034] running on GCE node
W20260927 18:34:50.756316 14614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.756704 14608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:50.756942 14608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:50.758098 14608 hybrid_clock.cc:648] HybridClock initialized: now 1790534090758092 us; error 35 us; skew 500 ppm
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:50.760582 14608 init.cc:377] Logged in from keytab as oryx/127.5.151.194@KRBTEST.COM (short username oryx)
I20260927 18:34:50.761039 14608 webserver.cc:460] Webserver started at http://127.5.151.194:42509/ using document root <none> and password file <none>
I20260927 18:34:50.761263 14608 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:50.761340 14608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:50.762658 14608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:34:50.763216 14624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.763377 14608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:50.763468 14608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "541d41a91b9d4dcba16c72e7d2e45afd"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "a12b5164a6114568aafa8059aa22d5e7"
server_key_iv: "9028f88d3fa5267022ee19e0dcd75776"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.763785 14608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:50.785488 14608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:50.788707 14608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:50.788878 14608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:50.789104 14608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:50.789433 14608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:50.789485 14608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.789527 14608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:50.789556 14608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.795272 14608 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:34321
I20260927 18:34:50.795321 14737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:34321 every 8 connection(s)
I20260927 18:34:50.795639 14608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:34:50.800536 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 14608
I20260927 18:34:50.800880 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:36639
--local_ip_for_outbound_sockets=127.5.151.195
--tserver_master_addrs=127.5.151.254:45915
--webserver_port=35717
--webserver_interface=127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:34:50.803078 14609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.796273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:57485 (local address 127.5.151.254:45915)
0927 18:34:50.796344 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.796346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.796519 (+ 173us) server_negotiation.cc:407] Connection header received
0927 18:34:50.796993 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.796994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.797005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.797073 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.798010 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.798481 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.799410 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.799588 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.799642 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.803022 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:57485: BlockingRecv error: recv got EOF from 127.5.151.194:57485 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:34:50.803150 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.796235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:57485)
0927 18:34:50.796395 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.796434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.796910 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.797151 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.797154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.797728 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.797931 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.797936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.798572 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.798572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.799334 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.799336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.799729 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.799730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.799843 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.800284 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.800292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.802957 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260927 18:34:50.803455 14738 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:45915 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.151.254:45915: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.808475 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.803685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:41531)
0927 18:34:50.803731 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.803741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.803782 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.803896 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.803897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.804048 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.804102 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.804104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.804695 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.804697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.805643 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.805645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.805966 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.805967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.806046 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.806417 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.806423 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.808375 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:34:50.808501 14609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.803741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:41531 (local address 127.5.151.254:45915)
0927 18:34:50.803776 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.803778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.803787 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:50.803833 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.803834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.803842 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.803884 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.804167 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.804623 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.805728 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.805894 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.805942 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.808454 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:41531: BlockingRecv error: recv got EOF from 127.5.151.194:41531 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.813302 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.808855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:40843)
0927 18:34:50.808920 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.808932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.808983 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.809123 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.809124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.809228 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.809255 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.809256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.810194 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.810194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.810734 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.810735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.811240 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.811241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.811275 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.811712 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.811716 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.813256 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:34:50.813445 14738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:34:50.813474 14609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.808937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:40843 (local address 127.5.151.254:45915)
0927 18:34:50.808984 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.808986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.809019 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:34:50.809043 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.809044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.809057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.809106 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.809324 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.810111 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.810811 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.811155 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.811244 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.813358 (+ 2114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:40843: BlockingRecv error: recv got EOF from 127.5.151.194:40843 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 18:34:50.883303 14741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:34:50.883497 14741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:34:50.883519 14741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:34:50.884559 14741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:34:50.884589 14741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:34:50.884631 14741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:34:50.886046 14741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/krb5kdc/oryx_127.5.151.195.keytab
--principal=oryx/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:36639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=35717
--tserver_master_addrs=127.5.151.254:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:34:50.886240 14741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:34:50.886404 14741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:34:50.891340 14746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:34:50.891342 14749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.891373 14741 server_base.cc:1034] running on GCE node
W20260927 18:34:50.891333 14747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:34:50.891708 14741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:34:50.891968 14741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:34:50.893112 14741 hybrid_clock.cc:648] HybridClock initialized: now 1790534090893102 us; error 31 us; skew 500 ppm
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534090, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:34:50.895514 14741 init.cc:377] Logged in from keytab as oryx/127.5.151.195@KRBTEST.COM (short username oryx)
I20260927 18:34:50.895972 14741 webserver.cc:460] Webserver started at http://127.5.151.195:35717/ using document root <none> and password file <none>
I20260927 18:34:50.896199 14741 fs_manager.cc:359] Metadata directory not provided
I20260927 18:34:50.896276 14741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:34:50.897549 14741 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:34:50.898056 14756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.898214 14741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:34:50.898308 14741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "af715081a1f44031b95b4b7d75dd7cf1"
format_stamp: "Formatted at 2026-09-27 18:34:50 on dist-test-slave-19nl"
server_key: "ca69e1f8cb2d6884f39b14bbcedb38f6"
server_key_iv: "eef1004b6b5aa7ac2471e0594096d90c"
server_key_version: "encryptionkey@0"
I20260927 18:34:50.898676 14741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:34:50.918850 14741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:34:50.920770 14741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:34:50.920925 14741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:34:50.921149 14741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:34:50.921450 14741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:34:50.921501 14741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.921535 14741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:34:50.921561 14741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:34:50.927016 14741 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:36639
I20260927 18:34:50.927062 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:36639 every 8 connection(s)
I20260927 18:34:50.927372 14741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:34:50.933871 14609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.928007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:41451 (local address 127.5.151.254:45915)
0927 18:34:50.928064 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.928066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.928242 (+ 176us) server_negotiation.cc:407] Connection header received
0927 18:34:50.928698 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.928699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.928710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.928759 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.929599 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.930060 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.931032 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.931201 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.931257 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.933820 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:41451: BlockingRecv error: recv got EOF from 127.5.151.195:41451 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:34:50.933928 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.927992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:41451)
0927 18:34:50.928141 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.928172 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.928618 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.928811 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.928815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.929357 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.929531 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.929536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.930130 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.930131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.930968 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.930970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.931271 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.931272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.931362 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.931754 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.931762 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.933749 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260927 18:34:50.934129 14870 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:45915 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.151.254:45915: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
I20260927 18:34:50.936407 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 14741
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.938279 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.934315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:55215)
0927 18:34:50.934361 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.934370 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.934407 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.934633 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.934633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.934742 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.934779 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.934780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.935305 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.935306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.935763 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.935764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.936030 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.936031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.936072 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.936451 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.936455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.938239 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:34:50.938377 14609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.934461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:55215 (local address 127.5.151.254:45915)
0927 18:34:50.934514 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.934515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.934522 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:50.934539 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.934539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.934572 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.934631 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.934835 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.935246 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.935820 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.935976 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.936020 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.938321 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:55215: BlockingRecv error: recv got EOF from 127.5.151.195:55215 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:50 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:50.942935 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.938773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:49195)
0927 18:34:50.938861 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:50.938871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:50.938927 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:50.939033 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:50.939034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:50.939128 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:50.939148 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.939149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.939838 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.939839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.940307 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:50.940308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.940773 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.940774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:50.940818 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.941135 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:50.941140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:50.942858 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 18:34:50.943019 14609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:50.938823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:49195 (local address 127.5.151.254:45915)
0927 18:34:50.938865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.938867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.938906 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:34:50.938973 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.938973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.938981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.939030 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.939204 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.939761 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.940378 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.940699 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.940789 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.942954 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:49195: BlockingRecv error: recv got EOF from 127.5.151.195:49195 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 18:34:50.943048 14870 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:34:50.944082 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:50.936824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.5.151.254:45915)
0927 18:34:50.936903 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:50.936906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:50.937017 (+ 111us) server_negotiation.cc:407] Connection header received
0927 18:34:50.937072 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:50.937072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:50.937084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:50.937142 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:50.937847 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.938360 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.939425 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:50.939589 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:50.939636 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:50.941009 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:34:50.941013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:34:50.941014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:34:50.941022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:34:50.942849 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:50.943359 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:50.943361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:50.943361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:50.943448 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:34:50.943601 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:34:50.943601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:34:50.943601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:34:50.943779 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:34:50.943866 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:34:50.943957 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:34:50.944031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 18:34:51 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:51 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:51.685075 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:51.680591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:44171)
0927 18:34:51.680755 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:51.680765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:51.680814 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:51.680930 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:51.680931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:51.681077 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:51.681122 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:51.681124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.681719 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.681719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:51.682267 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:51.682269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.682798 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.682799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:51.682860 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:51.683263 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:51.683267 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:51.685015 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
W20260927 18:34:51.685207 14875 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:51.680704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:44171 (local address 127.5.151.254:45915)
0927 18:34:51.680820 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:51.680822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:51.680832 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:34:51.680852 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:51.680853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:51.680861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:51.680916 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:51.681181 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.681654 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.682359 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.682722 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.682790 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:51.685096 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:44171: BlockingRecv error: recv got EOF from 127.5.151.193:44171 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
Sep 27 18:34:51 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:51 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:51.823706 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:51.813831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:60879)
0927 18:34:51.813985 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:51.813996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:51.814051 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:51.814175 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:51.814176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:51.814334 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:51.814381 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:51.814383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.815006 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.815006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:51.815551 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:51.815552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.815908 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.815909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:51.815948 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:51.822019 (+ 6071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:51.822023 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:51.823643 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
W20260927 18:34:51.823828 14875 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:51.813880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:60879 (local address 127.5.151.254:45915)
0927 18:34:51.813951 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:51.813953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:51.814051 (+ 98us) server_negotiation.cc:407] Connection header received
0927 18:34:51.814106 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:51.814106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:51.814117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:51.814157 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:51.814441 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.814942 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.815621 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.815844 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.815897 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:51.823734 (+ 7837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:60879: BlockingRecv error: recv got EOF from 127.5.151.194:60879 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 18:34:51 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:51 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:51.947898 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:51.943360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:51617)
0927 18:34:51.943517 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:51.943529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:51.943613 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:51.943736 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:51.943736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:51.943905 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:51.943948 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:51.943950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.944575 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.944576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:51.945098 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:51.945100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.945457 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.945458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:51.945519 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:51.946007 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:51.946020 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:51.947837 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
W20260927 18:34:51.948006 14875 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:51.943455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:51617 (local address 127.5.151.254:45915)
0927 18:34:51.943519 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:51.943521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:51.943594 (+ 73us) server_negotiation.cc:407] Connection header received
0927 18:34:51.943667 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:51.943668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:51.943681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:51.943731 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:51.944024 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.944495 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.945181 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:51.945374 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:51.945456 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:51.947916 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:51617: BlockingRecv error: recv got EOF from 127.5.151.195:51617 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 18:34:52 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:52 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:52.690025 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:52.685505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:34073)
0927 18:34:52.685651 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:52.685660 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:52.685703 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:52.685868 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:52.685869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:52.686033 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:52.686079 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:52.686081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.686720 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.686721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:52.687237 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:52.687239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.687529 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.687530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:52.687567 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:52.687958 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:52.687962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:52.689932 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260927 18:34:52.690076 14879 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:52.685588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:34073 (local address 127.5.151.254:45915)
0927 18:34:52.685710 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:52.685712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:52.685721 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:52.685788 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:52.685789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:52.685799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:52.685850 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:52.686143 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.686636 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.687305 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.687471 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.687516 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:52.690008 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:34073: BlockingRecv error: recv got EOF from 127.5.151.193:34073 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
Sep 27 18:34:52 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:52 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:52.828970 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:52.824135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:58285)
0927 18:34:52.824308 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:52.824320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:52.824411 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:52.824586 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:52.824587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:52.824758 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:52.824807 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:52.824809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.825447 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.825448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:52.825952 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:52.825954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.826312 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.826313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:52.826380 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:52.826918 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:52.826923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:52.828914 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
W20260927 18:34:52.829066 14879 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:52.824223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:58285 (local address 127.5.151.254:45915)
0927 18:34:52.824295 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:52.824298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:52.824389 (+ 91us) server_negotiation.cc:407] Connection header received
0927 18:34:52.824485 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:52.824486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:52.824499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:52.824579 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:52.824885 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.825369 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.826032 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.826224 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.826295 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:52.829001 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:58285: BlockingRecv error: recv got EOF from 127.5.151.194:58285 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 18:34:52 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:52 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:52.952451 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:52.948279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:56281)
0927 18:34:52.948405 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:52.948430 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:52.948476 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:52.948633 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:52.948633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:52.948812 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:52.948852 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:52.948853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.949455 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.949456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:52.949929 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:52.949930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.950293 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.950294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:52.950354 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:52.950748 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:52.950752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:52.952364 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
W20260927 18:34:52.952485 14879 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:52.948390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:56281 (local address 127.5.151.254:45915)
0927 18:34:52.948455 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:52.948457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:52.948478 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:34:52.948533 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:52.948534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:52.948546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:52.948607 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:52.948929 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.949381 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.950006 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:52.950204 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:52.950285 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:52.952431 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:56281: BlockingRecv error: recv got EOF from 127.5.151.195:56281 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 18:34:53 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:53 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:53.695230 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:53.690527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:35707)
0927 18:34:53.690704 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:53.690717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:53.690790 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:53.690922 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:53.690923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:53.691064 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:53.691111 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:53.691112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.691692 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.691693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:53.692191 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:53.692193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.692510 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.692510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:53.692548 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:53.693030 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:53.693035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:53.695120 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
W20260927 18:34:53.695272 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:53.690634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:35707 (local address 127.5.151.254:45915)
0927 18:34:53.690791 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:53.690793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:53.690801 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:53.690844 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:53.690845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:53.690854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:53.690918 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:53.691167 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.691632 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.692259 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.692448 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.692499 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:53.695201 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:35707: BlockingRecv error: recv got EOF from 127.5.151.193:35707 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
Sep 27 18:34:53 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:53 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:53.834056 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:53.829493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:48243)
0927 18:34:53.829620 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:53.829630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:53.829680 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:53.829815 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:53.829816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:53.829975 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:53.830022 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:53.830024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.830711 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.830712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:53.831222 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:53.831224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.831570 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.831570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:53.831610 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:53.832185 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:53.832190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:53.834002 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
W20260927 18:34:53.834162 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:53.829561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:48243 (local address 127.5.151.254:45915)
0927 18:34:53.829652 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:53.829654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:53.829667 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:34:53.829735 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:53.829736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:53.829746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:53.829801 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:53.830082 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.830643 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.831304 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.831503 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.831562 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:53.834090 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:48243: BlockingRecv error: recv got EOF from 127.5.151.194:48243 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 18:34:53 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:53 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:53.957350 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:53.952986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:59907)
0927 18:34:53.953116 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:53.953127 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:53.953191 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:53.953320 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:53.953321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:53.953477 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:53.953524 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:53.953525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.954165 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.954166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:53.954751 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:53.954753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.955079 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.955080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:53.955119 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:53.955584 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:53.955589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:53.957283 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
W20260927 18:34:53.957402 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:53.953048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:59907 (local address 127.5.151.254:45915)
0927 18:34:53.953129 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:53.953131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:53.953197 (+ 66us) server_negotiation.cc:407] Connection header received
0927 18:34:53.953236 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:53.953237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:53.953247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:53.953297 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:53.953587 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.954090 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.954835 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:53.955014 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:53.955069 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:53.957358 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:59907: BlockingRecv error: recv got EOF from 127.5.151.195:59907 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 18:34:54 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:54 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:54.699950 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:54.695750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:49291)
0927 18:34:54.695916 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:54.695929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:54.695992 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:54.696179 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:54.696180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:54.696376 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:54.696450 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:54.696453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.697038 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.697039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:54.697511 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:54.697512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.697791 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.697792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:54.697826 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:54.698252 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:54.698256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:54.699899 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260927 18:34:54.700083 14887 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:54.695842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:49291 (local address 127.5.151.254:45915)
0927 18:34:54.695965 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:54.695967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:54.695974 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:54.696064 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:54.696065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:54.696093 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:54.696158 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:54.696529 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.696978 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.697575 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.697736 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.697781 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:54.700000 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:49291: BlockingRecv error: recv got EOF from 127.5.151.193:49291 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
Sep 27 18:34:54 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:54 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:54.839004 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:54.834679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:43469)
0927 18:34:54.834805 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:54.834815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:54.834872 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:54.835041 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:54.835043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:54.835208 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:54.835275 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:54.835277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.835877 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.835877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:54.836373 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:54.836374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.836662 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.836662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:54.836699 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:54.837117 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:54.837121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:54.838941 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
W20260927 18:34:54.839097 14887 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:54.834710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:43469 (local address 127.5.151.254:45915)
0927 18:34:54.834759 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:54.834761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:54.834864 (+ 103us) server_negotiation.cc:407] Connection header received
0927 18:34:54.834920 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:54.834921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:54.834934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:54.835001 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:54.835348 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.835809 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.836435 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.836603 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.836650 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:54.839031 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:43469: BlockingRecv error: recv got EOF from 127.5.151.194:43469 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 18:34:54 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:54 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:54.962076 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:54.957801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:44669)
0927 18:34:54.957923 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:54.957933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:54.958003 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:54.958130 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:54.958131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:54.958281 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:54.958329 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:54.958330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.958998 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.958999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:54.959491 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:54.959492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.959826 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.959826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:54.959867 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:54.960271 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:54.960276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:54.962021 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
W20260927 18:34:54.962181 14887 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:54.957885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:44669 (local address 127.5.151.254:45915)
0927 18:34:54.957933 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:54.957935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:54.957991 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:34:54.958055 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:54.958056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:54.958065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:54.958105 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:54.958389 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.958930 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.959583 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:54.959753 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:54.959817 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:54.962111 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:44669: BlockingRecv error: recv got EOF from 127.5.151.195:44669 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 18:34:55 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:55 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:55.705338 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:55.700416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:35631)
0927 18:34:55.700550 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:55.700563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:55.700629 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:55.700792 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:55.700793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:55.700978 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:55.701021 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:55.701023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.701723 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.701724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:55.702279 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:55.702282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.702863 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.702863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:55.702908 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:55.703416 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:55.703420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:55.705283 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260927 18:34:55.705560 14891 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:55.700494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:35631 (local address 127.5.151.254:45915)
0927 18:34:55.700601 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:55.700603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:55.700611 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:34:55.700684 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:55.700685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:55.700696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:55.700773 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:55.701108 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.701646 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.702363 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.702775 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.702849 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:55.705467 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:35631: BlockingRecv error: recv got EOF from 127.5.151.193:35631 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
Sep 27 18:34:55 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:55 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:55.844187 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:55.839500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:49631)
0927 18:34:55.839640 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:55.839666 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:55.839724 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:55.839879 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:55.839879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:55.840051 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:55.840100 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:55.840102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.840896 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.840897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:55.841508 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:55.841510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.841880 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.841881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:55.841921 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:55.842407 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:55.842413 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:55.844131 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260927 18:34:55.844300 14891 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:55.839636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:49631 (local address 127.5.151.254:45915)
0927 18:34:55.839698 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:55.839700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:55.839709 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:34:55.839810 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:55.839810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:55.839822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:55.839863 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:55.840185 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.840786 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.841608 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.841796 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.841858 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:55.844224 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:49631: BlockingRecv error: recv got EOF from 127.5.151.194:49631 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 18:34:55 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:55 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:55.966923 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:55.962556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:60657)
0927 18:34:55.962710 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:55.962724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:55.962782 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:55.962884 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:55.962885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:55.963031 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:55.963104 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:55.963105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.963689 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.963690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:55.964194 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:55.964195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.964535 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.964536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:55.964596 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:55.965014 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:55.965018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:55.966862 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260927 18:34:55.967007 14891 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:55.962637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:60657 (local address 127.5.151.254:45915)
0927 18:34:55.962719 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:55.962721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:55.962778 (+ 57us) server_negotiation.cc:407] Connection header received
0927 18:34:55.962823 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:55.962823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:55.962835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:55.962881 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:55.963156 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.963627 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.964280 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:55.964464 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:55.964517 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:55.966945 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:60657: BlockingRecv error: recv got EOF from 127.5.151.195:60657 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 18:34:56 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:56 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:56.710151 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:56.705757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:34151)
0927 18:34:56.705906 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:56.705916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:56.705967 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:56.706097 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:56.706097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:56.706239 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:56.706283 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:56.706285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.707050 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.707051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:56.707555 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:56.707557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.707858 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.707859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:56.707898 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:56.708305 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:56.708309 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:56.710075 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
W20260927 18:34:56.710259 14895 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:56.705792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:34151 (local address 127.5.151.254:45915)
0927 18:34:56.705937 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:56.705939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:56.705957 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:34:56.706018 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:56.706018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:56.706028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:56.706083 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:56.706341 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.706984 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.707617 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.707794 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.707845 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:56.710168 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:34151: BlockingRecv error: recv got EOF from 127.5.151.193:34151 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
Sep 27 18:34:56 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:56 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:56.853813 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:56.844641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:44853)
0927 18:34:56.844801 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:56.844816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:56.844890 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:56.845067 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:56.845068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:56.845236 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:56.845282 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:56.845284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.845937 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.845938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:56.846486 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:56.846488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.846847 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.846848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:56.846904 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:56.851914 (+ 5010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:56.851919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:56.853757 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
W20260927 18:34:56.853941 14895 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:56.844719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:44853 (local address 127.5.151.254:45915)
0927 18:34:56.844786 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:56.844788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:56.844870 (+ 82us) server_negotiation.cc:407] Connection header received
0927 18:34:56.844983 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:56.844984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:56.844995 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:56.845062 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:56.845365 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.845849 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.846578 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.846767 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.846847 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:56.853856 (+ 7009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:44853: BlockingRecv error: recv got EOF from 127.5.151.194:44853 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 18:34:56 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:56 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:56.971882 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:56.967338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:54477)
0927 18:34:56.967494 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:56.967503 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:56.967556 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:56.967663 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:56.967664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:56.967812 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:56.967860 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:56.967862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.968505 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.968506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:56.969075 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:56.969078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.969411 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.969412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:56.969460 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:56.969970 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:56.969975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:56.971805 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
W20260927 18:34:56.972011 14895 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:56.967379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:54477 (local address 127.5.151.254:45915)
0927 18:34:56.967428 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:56.967430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:56.967546 (+ 116us) server_negotiation.cc:407] Connection header received
0927 18:34:56.967598 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:56.967599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:56.967608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:56.967652 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:56.967920 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.968416 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.969176 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:56.969354 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:56.969402 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:56.971904 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:54477: BlockingRecv error: recv got EOF from 127.5.151.195:54477 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 18:34:57 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:57 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:57.715157 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:57.710762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:50039)
0927 18:34:57.710887 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:57.710897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:57.710955 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:57.711122 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:57.711123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:57.711273 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:57.711320 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:57.711321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.711932 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.711932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:57.712429 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:57.712430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.712730 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.712731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:57.712768 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:57.713168 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:57.713172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:57.715099 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260927 18:34:57.715260 14899 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:57.710852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:50039 (local address 127.5.151.254:45915)
0927 18:34:57.711004 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:57.711006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:57.711013 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:57.711034 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:57.711035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:57.711044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:57.711104 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:57.711381 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.711856 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.712507 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.712673 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.712719 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:57.715187 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:50039: BlockingRecv error: recv got EOF from 127.5.151.193:50039 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
Sep 27 18:34:57 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:57 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:57.858649 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:57.854294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:40521)
0927 18:34:57.854468 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:57.854479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:57.854537 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:57.854729 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:57.854730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:57.854887 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:57.854935 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:57.854936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.855550 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.855551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:57.856054 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:57.856056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.856352 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.856353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:57.856389 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:57.856814 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:57.856818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:57.858518 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
W20260927 18:34:57.858700 14899 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:57.854351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:40521 (local address 127.5.151.254:45915)
0927 18:34:57.854415 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:57.854417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:57.854554 (+ 137us) server_negotiation.cc:407] Connection header received
0927 18:34:57.854645 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:57.854646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:57.854663 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:57.854710 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:57.854997 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.855475 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.856121 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.856292 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.856341 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:57.858656 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:40521: BlockingRecv error: recv got EOF from 127.5.151.194:40521 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 18:34:57 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:57 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:57.976828 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:57.972313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:51977)
0927 18:34:57.972465 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:57.972475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:57.972534 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:57.972651 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:57.972652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:57.972812 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:57.972859 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:57.972860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.973495 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.973496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:57.974035 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:57.974036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.974337 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.974337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:57.974376 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:57.974906 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:57.974911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:57.976732 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
W20260927 18:34:57.976864 14899 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:57.972365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:51977 (local address 127.5.151.254:45915)
0927 18:34:57.972441 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:57.972443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:57.972534 (+ 91us) server_negotiation.cc:407] Connection header received
0927 18:34:57.972574 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:57.972575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:57.972585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:57.972633 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:57.972916 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.973394 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.974106 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:57.974274 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:57.974323 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:57.976808 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:51977: BlockingRecv error: recv got EOF from 127.5.151.195:51977 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 18:34:58 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:58 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:58.720346 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:58.715621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:42181)
0927 18:34:58.715754 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:58.715765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:58.715818 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:58.715973 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:58.715974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:58.716126 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:58.716174 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:58.716175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.716864 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.716866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:58.717551 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:58.717553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.717883 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.717884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:58.717924 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:58.718342 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:58.718346 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:58.720242 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260927 18:34:58.720386 14903 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:58.715739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:42181 (local address 127.5.151.254:45915)
0927 18:34:58.715858 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:58.715859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:58.715866 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:34:58.715887 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:58.715887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:58.715897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:58.715954 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:58.716234 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.716792 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.717620 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.717820 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.717871 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:58.720335 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:42181: BlockingRecv error: recv got EOF from 127.5.151.193:42181 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
Sep 27 18:34:58 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:58 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:58.863495 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:58.859089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:58553)
0927 18:34:58.859222 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:58.859234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:58.859293 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:58.859432 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:58.859433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:58.859600 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:58.859649 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:58.859650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.860407 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.860408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:58.860926 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:58.860927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.861242 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.861243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:58.861281 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:58.861706 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:58.861710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:58.863439 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260927 18:34:58.863610 14903 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:58.859151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:58553 (local address 127.5.151.254:45915)
0927 18:34:58.859215 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:58.859217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:58.859295 (+ 78us) server_negotiation.cc:407] Connection header received
0927 18:34:58.859356 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:58.859357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:58.859368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:58.859410 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:58.859711 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.860341 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.860995 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.861179 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.861228 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:58.863540 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:58553: BlockingRecv error: recv got EOF from 127.5.151.194:58553 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 18:34:58 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:58 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:58.982169 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:58.977301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:60267)
0927 18:34:58.977412 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:58.977424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:58.977477 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:58.977646 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:58.977647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:58.977809 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:58.977858 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:58.977859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.978495 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.978496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:58.979060 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:58.979061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.979387 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.979388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:58.979426 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:58.979929 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:58.979934 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:58.982105 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
W20260927 18:34:58.982278 14903 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:58.977373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:60267 (local address 127.5.151.254:45915)
0927 18:34:58.977456 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:58.977457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:58.977478 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:34:58.977542 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:58.977542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:58.977554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:58.977597 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:58.977940 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.978422 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.979131 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:58.979312 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:58.979375 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:58.982184 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:60267: BlockingRecv error: recv got EOF from 127.5.151.195:60267 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 18:34:59 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:59 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:59.725214 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:59.720806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:41167)
0927 18:34:59.720953 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:59.720964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:59.721018 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:59.721144 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:59.721144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:59.721303 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:59.721351 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:59.721353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.722005 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.722006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:59.722515 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:59.722517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.722935 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.722936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:59.723010 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:59.723420 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:59.723425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:59.725157 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260927 18:34:59.725379 14907 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:59.720895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:41167 (local address 127.5.151.254:45915)
0927 18:34:59.721003 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:59.721005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:59.721019 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:34:59.721064 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:59.721064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:59.721074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:59.721130 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:59.721416 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.721898 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.722672 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.722857 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.722934 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:59.725254 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:41167: BlockingRecv error: recv got EOF from 127.5.151.193:41167 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
Sep 27 18:34:59 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:59 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:59.868400 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:59.863983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:51049)
0927 18:34:59.864113 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:59.864125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:59.864187 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:59.864302 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:59.864303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:59.864456 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:59.864505 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:59.864507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.865121 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.865122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:59.865637 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:59.865638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.865937 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.865938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:59.865976 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:59.866511 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:59.866515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:59.868335 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":93,"threads_started":1}
W20260927 18:34:59.868466 14907 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:59.864045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:51049 (local address 127.5.151.254:45915)
0927 18:34:59.864087 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:59.864089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:59.864171 (+ 82us) server_negotiation.cc:407] Connection header received
0927 18:34:59.864226 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:59.864226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:59.864237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:59.864286 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:59.864564 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.865058 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.865700 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.865877 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.865926 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:59.868420 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:51049: BlockingRecv error: recv got EOF from 127.5.151.194:51049 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 18:34:59 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:34:59 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:34:59.988282 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:34:59.982700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:48645)
0927 18:34:59.982837 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:34:59.982848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:34:59.982918 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:34:59.983093 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:34:59.983094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:34:59.983256 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:34:59.983305 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:59.983306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.984138 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.984140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:59.984781 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:34:59.984784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.985283 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.985284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:34:59.985356 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:59.985883 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:34:59.985889 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:34:59.988202 (+ 2313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
W20260927 18:34:59.988392 14907 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:34:59.982765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:48645 (local address 127.5.151.254:45915)
0927 18:34:59.982834 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 18:34:59.982836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:34:59.982902 (+ 66us) server_negotiation.cc:407] Connection header received
0927 18:34:59.982981 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:34:59.982981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:34:59.982997 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:34:59.983071 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:34:59.983392 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.984037 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.984874 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:34:59.985163 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:34:59.985227 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:34:59.988298 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:48645: BlockingRecv error: recv got EOF from 127.5.151.195:48645 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 18:35:00 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:00 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:00.730769 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:00.725720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:58165)
0927 18:35:00.725882 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:00.725893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:00.725953 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:00.726148 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:00.726149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:00.726334 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:00.726381 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:00.726383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.727150 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.727151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:00.727799 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:00.727802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.728207 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.728209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:00.728267 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:00.728676 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:00.728680 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:00.730669 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
W20260927 18:35:00.730824 14911 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:00.725762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:58165 (local address 127.5.151.254:45915)
0927 18:35:00.726033 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:00.726034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:00.726040 (+ 6us) server_negotiation.cc:407] Connection header received
0927 18:35:00.726058 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:00.726059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:00.726068 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:00.726140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:00.726480 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.727067 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.727882 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.728136 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.728210 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:00.730766 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:58165: BlockingRecv error: recv got EOF from 127.5.151.193:58165 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":53,"threads_started":1}
Sep 27 18:35:00 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:00 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:00.873378 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:00.868863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:59313)
0927 18:35:00.869000 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:00.869010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:00.869071 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:00.869259 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:00.869260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:00.869455 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:00.869499 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:00.869501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.870235 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.870236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:00.870780 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:00.870781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.871151 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.871152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:00.871209 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:00.871623 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:00.871627 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:00.873287 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260927 18:35:00.873425 14911 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:00.868904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:59313 (local address 127.5.151.254:45915)
0927 18:35:00.868969 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:00.868972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:00.869071 (+ 99us) server_negotiation.cc:407] Connection header received
0927 18:35:00.869154 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:00.869155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:00.869183 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:00.869254 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:00.869589 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.870152 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.870862 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.871067 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.871151 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:00.873363 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:59313: BlockingRecv error: recv got EOF from 127.5.151.194:59313 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 18:35:00 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:00 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:00.993204 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:00.988712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:34511)
0927 18:35:00.988841 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:00.988852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:00.988909 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:00.989085 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:00.989086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:00.989271 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:00.989316 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:00.989318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.989959 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.989961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:00.990472 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:00.990473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.990866 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.990867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:00.990922 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:00.991400 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:00.991405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:00.993150 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260927 18:35:00.993305 14911 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:00.988856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:34511 (local address 127.5.151.254:45915)
0927 18:35:00.988905 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:00.988907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:00.988916 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:35:00.988969 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:00.988970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:00.988983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:00.989074 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:00.989396 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.989873 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.990600 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:00.990783 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:00.990865 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:00.993224 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:34511: BlockingRecv error: recv got EOF from 127.5.151.195:34511 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 18:35:01 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:01 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:01.735836 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:01.731211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:58763)
0927 18:35:01.731353 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:01.731363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:01.731409 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:01.731537 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:01.731538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:01.731698 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:01.731743 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:01.731745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.732415 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.732415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:01.732949 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:01.732951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.733359 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.733360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:01.733410 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:01.733912 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:01.733917 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:01.735765 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260927 18:35:01.735939 14915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:01.731283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:58763 (local address 127.5.151.254:45915)
0927 18:35:01.731413 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:01.731414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:01.731424 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:35:01.731447 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:01.731447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:01.731456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:01.731517 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:01.731825 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.732342 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.733052 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.733269 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.733325 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:01.735865 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:58763: BlockingRecv error: recv got EOF from 127.5.151.193:58763 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Sep 27 18:35:01 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:01 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:01.881512 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:01.873881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:46411)
0927 18:35:01.874042 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:01.874053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:01.874139 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:01.874286 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:01.874287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:01.874461 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:01.874502 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:01.874504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.875145 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.875147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:01.875644 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:01.875646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.875978 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.875979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:01.876032 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:01.879599 (+ 3567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:01.879603 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:01.881451 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
W20260927 18:35:01.881604 14915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:01.873967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:46411 (local address 127.5.151.254:45915)
0927 18:35:01.874035 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:01.874037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:01.874131 (+ 94us) server_negotiation.cc:407] Connection header received
0927 18:35:01.874201 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:01.874201 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:01.874213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:01.874281 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:01.874591 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.875052 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.875721 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.875900 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.875978 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:01.881542 (+ 5564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:46411: BlockingRecv error: recv got EOF from 127.5.151.194:46411 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 18:35:01 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:01 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:01.998361 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:01.993659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:54899)
0927 18:35:01.993840 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:01.993852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:01.993931 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:01.994097 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:01.994098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:01.994288 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:01.994330 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:01.994332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.995065 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.995066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:01.995622 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:01.995624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.996031 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.996033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:01.996089 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:01.996519 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:01.996523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:01.998290 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
W20260927 18:35:01.998430 14915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:01.993745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:54899 (local address 127.5.151.254:45915)
0927 18:35:01.993832 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:01.993834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:01.993925 (+ 91us) server_negotiation.cc:407] Connection header received
0927 18:35:01.993994 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:01.993995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:01.994009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:01.994084 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:01.994416 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.994983 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.995721 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:01.995971 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:01.996034 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:01.998382 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:54899: BlockingRecv error: recv got EOF from 127.5.151.195:54899 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 18:35:02 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:02 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:02.741170 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:02.736400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:42161)
0927 18:35:02.736561 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:02.736572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:02.736631 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:02.736765 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:02.736766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:02.736938 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:02.736983 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:02.736985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.737642 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.737643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:02.738452 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:02.738455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.738805 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.738805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:02.738844 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:02.739254 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:02.739258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:02.741076 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
W20260927 18:35:02.741225 14919 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:02.736498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:42161 (local address 127.5.151.254:45915)
0927 18:35:02.736598 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:02.736600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:02.736612 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:35:02.736685 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:02.736685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:02.736695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:02.736750 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:02.737062 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.737548 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.738514 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.738747 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.738799 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:02.741156 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:42161: BlockingRecv error: recv got EOF from 127.5.151.193:42161 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":49,"threads_started":1}
Sep 27 18:35:02 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:02 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:02.885912 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:02.881921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:53299)
0927 18:35:02.882035 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:02.882051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:02.882110 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:02.882232 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:02.882233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:02.882388 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:02.882433 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:02.882434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.883029 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.883030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:02.883539 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:02.883541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.883833 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.883834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:02.883871 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:02.884293 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:02.884297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:02.885850 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
W20260927 18:35:02.886032 14919 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:02.881993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:53299 (local address 127.5.151.254:45915)
0927 18:35:02.882062 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:02.882066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:02.882108 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:35:02.882157 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:02.882157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:02.882167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:02.882213 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:02.882488 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.882970 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.883604 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.883775 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:02.883822 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:02.885938 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:53299: BlockingRecv error: recv got EOF from 127.5.151.194:53299 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 18:35:02 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:03 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:03.003191 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:02.998914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:59347)
0927 18:35:02.999041 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:02.999052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:02.999120 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:02.999238 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:02.999239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:02.999404 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:02.999447 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:02.999448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.000048 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.000050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:03.000555 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:03.000557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.000849 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.000850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:03.000889 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:03.001362 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:03.001366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:03.003123 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
W20260927 18:35:03.003249 14919 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:02.998971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:59347 (local address 127.5.151.254:45915)
0927 18:35:02.999029 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:02.999031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:02.999119 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:35:02.999162 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:02.999163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:02.999173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:02.999223 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:02.999503 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:02.999969 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.000622 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.000790 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.000837 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:03.003204 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:59347: BlockingRecv error: recv got EOF from 127.5.151.195:59347 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 18:35:03 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:03 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:03.746027 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:03.741701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:60255)
0927 18:35:03.741841 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:03.741851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:03.741918 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:03.742086 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:03.742087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:03.742249 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:03.742296 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:03.742297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.742919 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.742920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:03.743433 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:03.743435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.743760 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.743761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:03.743801 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:03.744222 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:03.744227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:03.745919 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260927 18:35:03.746063 14923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:03.741773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:60255 (local address 127.5.151.254:45915)
0927 18:35:03.741891 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:03.741893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:03.741903 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:35:03.741997 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:03.741998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:03.742010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:03.742070 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:03.742361 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.742858 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.743504 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.743690 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.743744 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:03.745993 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:60255: BlockingRecv error: recv got EOF from 127.5.151.193:60255 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
Sep 27 18:35:03 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:03 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:03.890919 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:03.886359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:50619)
0927 18:35:03.886490 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:03.886500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:03.886544 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:03.886783 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:03.886785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:03.886963 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:03.887010 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:03.887012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.887685 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.887686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:03.888198 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:03.888200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.888553 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.888554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:03.888611 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:03.889041 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:03.889045 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:03.890830 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
W20260927 18:35:03.891021 14923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:03.886597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:50619 (local address 127.5.151.254:45915)
0927 18:35:03.886670 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:03.886671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:03.886678 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:35:03.886700 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:03.886700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:03.886710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:03.886772 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:03.887084 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.887580 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.888278 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:03.888470 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:03.888567 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:03.890952 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:50619: BlockingRecv error: recv got EOF from 127.5.151.194:50619 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 18:35:04 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:04 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:04.007782 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:04.003617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:33609)
0927 18:35:04.003758 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:04.003768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:04.003819 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:04.003930 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:04.003930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:04.004084 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:04.004131 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:04.004132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.004728 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.004729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:04.005238 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:04.005239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.005525 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.005526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:04.005564 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:04.005965 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:04.005970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:04.007717 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260927 18:35:04.007844 14923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:04.003675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:33609 (local address 127.5.151.254:45915)
0927 18:35:04.003722 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:04.003724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:04.003818 (+ 94us) server_negotiation.cc:407] Connection header received
0927 18:35:04.003858 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:04.003858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:04.003869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:04.003916 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:04.004197 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.004670 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.005301 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.005467 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.005514 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:04.007799 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:33609: BlockingRecv error: recv got EOF from 127.5.151.195:33609 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 18:35:04 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:04 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:04.750758 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:04.746482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:55399)
0927 18:35:04.746610 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:04.746622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:04.746683 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:04.746824 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:04.746825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:04.746984 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:04.747028 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:04.747030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.747615 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.747616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:04.748132 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:04.748134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.748441 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.748442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:04.748491 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:04.748933 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:04.748937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:04.750703 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":95,"threads_started":1}
W20260927 18:35:04.750903 14927 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:04.746569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:55399 (local address 127.5.151.254:45915)
0927 18:35:04.746704 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:04.746706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:04.746715 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:35:04.746740 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:04.746740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:04.746750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:04.746806 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:04.747102 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.747555 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.748207 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.748383 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.748430 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:04.750815 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:55399: BlockingRecv error: recv got EOF from 127.5.151.193:55399 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
Sep 27 18:35:04 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:04 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:04.895890 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:04.891444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:44091)
0927 18:35:04.891586 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:04.891597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:04.891659 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:04.891783 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:04.891784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:04.891948 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:04.891997 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:04.891999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.892644 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.892645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:04.893187 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:04.893189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.893563 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.893565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:04.893630 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:04.894062 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:04.894067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:04.895784 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260927 18:35:04.895938 14927 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:04.891495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:44091 (local address 127.5.151.254:45915)
0927 18:35:04.891565 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:04.891566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:04.891660 (+ 94us) server_negotiation.cc:407] Connection header received
0927 18:35:04.891706 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:04.891707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:04.891717 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:04.891768 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:04.892061 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.892553 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.893275 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:04.893475 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:04.893550 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:04.895878 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:44091: BlockingRecv error: recv got EOF from 127.5.151.194:44091 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 18:35:05 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:05 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.195@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:05.012650 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:05.008214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.195:57175)
0927 18:35:05.008379 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:05.008391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:05.008481 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:05.008626 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:05.008626 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:05.008803 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:05.008847 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:05.008849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.009488 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.009490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:05.010014 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:05.010016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.010370 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.010370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:05.010435 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:05.010844 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:05.010848 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:05.012595 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
W20260927 18:35:05.012791 14927 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:05.008309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:57175 (local address 127.5.151.254:45915)
0927 18:35:05.008378 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:05.008380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:05.008463 (+ 83us) server_negotiation.cc:407] Connection header received
0927 18:35:05.008535 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:05.008536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:05.008548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:05.008619 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:05.008926 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.009408 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.010102 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.010288 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.010368 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:05.012710 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.195:57175: BlockingRecv error: recv got EOF from 127.5.151.195:57175 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 18:35:05 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:05 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.193@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:05.755369 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:05.751253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.193:53811)
0927 18:35:05.751374 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:05.751383 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:05.751431 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:05.751582 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:05.751583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:05.751735 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:05.751782 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:05.751784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.752394 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.752394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:05.752888 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:05.752889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.753194 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.753194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:05.753230 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:05.753632 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:05.753637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:05.755282 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
W20260927 18:35:05.755463 14931 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:05.751308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:53811 (local address 127.5.151.254:45915)
0927 18:35:05.751432 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:05.751434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:05.751485 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:35:05.751504 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:05.751505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:05.751514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:05.751560 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:05.751845 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.752332 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.752952 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.753129 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.753177 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:05.755382 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.193:53811: BlockingRecv error: recv got EOF from 127.5.151.193:53811 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
Sep 27 18:35:05 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:35:05 dist-test-slave-19nl krb5kdc[13873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.151.194@KRBTEST.COM for oryx/127.5.151.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:35:05.901479 14932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:05.896394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45915 (local address 127.5.151.194:59239)
0927 18:35:05.896569 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:05.896580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:05.896669 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:05.896849 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:05.896851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:05.897071 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:05.897119 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:05.897121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.897816 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.897817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:05.898375 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:05.898376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.898810 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.898812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:05.898867 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:05.899437 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:05.899442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:05.901371 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.151.254:45915: Server oryx/127.5.151.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
W20260927 18:35:05.901513 14931 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:05.896482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:59239 (local address 127.5.151.254:45915)
0927 18:35:05.896550 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:05.896552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:05.896646 (+ 94us) server_negotiation.cc:407] Connection header received
0927 18:35:05.896761 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:05.896762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:05.896774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:05.896844 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:05.897199 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.897719 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.898460 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:05.898728 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:05.898809 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:05.901448 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.151.194:59239: BlockingRecv error: recv got EOF from 127.5.151.194:59239 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:35:05.942030 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 14456
I20260927 18:35:05.946203 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 14608
I20260927 18:35:05.950073 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 14741
I20260927 18:35:05.954109 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 14386
2026-09-27T18:35:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17413 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T18:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:05Z Disabled control of system clock
I20260927 18:35:05.980402 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:33721
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:35:06.060869 14936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:06.061056 14936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:06.061075 14936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:06.062136 14936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:06.062170 14936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:06.062181 14936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:06.062193 14936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:06.062211 14936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:06.063432 14936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:33721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:06.063606 14936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:06.063819 14936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:06.068638 14944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.068595 14942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.068646 14936 server_base.cc:1034] running on GCE node
W20260927 18:35:06.068614 14941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.069129 14936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:06.069442 14936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:06.070613 14936 hybrid_clock.cc:648] HybridClock initialized: now 1790534106070599 us; error 43 us; skew 500 ppm
I20260927 18:35:06.071782 14936 webserver.cc:460] Webserver started at http://127.5.151.254:39585/ using document root <none> and password file <none>
I20260927 18:35:06.072026 14936 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:06.072094 14936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:06.072278 14936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:06.073282 14936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "5fdb40955d15407781df0d895678b2c4"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "e72a9a7f11e6775b12fd3f04eeeb1bf8"
server_key_iv: "ba69b30697cfd1c94072374db8868d1e"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.073649 14936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "5fdb40955d15407781df0d895678b2c4"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "e72a9a7f11e6775b12fd3f04eeeb1bf8"
server_key_iv: "ba69b30697cfd1c94072374db8868d1e"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.074976 14936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:35:06.075659 14950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.075815 14936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:35:06.075907 14936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "5fdb40955d15407781df0d895678b2c4"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "e72a9a7f11e6775b12fd3f04eeeb1bf8"
server_key_iv: "ba69b30697cfd1c94072374db8868d1e"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.075989 14936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:06.083683 14936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:06.083957 14936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:06.084097 14936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:06.088014 14936 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:33721
I20260927 18:35:06.088042 15002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:33721 every 8 connection(s)
I20260927 18:35:06.088361 14936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:06.088961 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:06.091410 15003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4: Bootstrap starting.
I20260927 18:35:06.091985 15003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:06.092291 15003 log.cc:824] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:06.092952 15003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4: No bootstrap required, opened a new log
I20260927 18:35:06.094169 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 14936
I20260927 18:35:06.094247 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:35:06.094458 5727 external_mini_cluster.cc:1428] Setting key cd00b0553bcc5d7138d7152ec4c131d2
I20260927 18:35:06.094614 15003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdb40955d15407781df0d895678b2c4" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 33721 } }
I20260927 18:35:06.094775 15003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:06.094811 15003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdb40955d15407781df0d895678b2c4, State: Initialized, Role: FOLLOWER
I20260927 18:35:06.094996 15003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdb40955d15407781df0d895678b2c4" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 33721 } }
I20260927 18:35:06.095106 15003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:06.095155 15003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:06.095224 15003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:06.095763 15003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdb40955d15407781df0d895678b2c4" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 33721 } }
I20260927 18:35:06.095870 15003 leader_election.cc:302] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fdb40955d15407781df0d895678b2c4; no voters:
I20260927 18:35:06.096040 15003 leader_election.cc:288] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:06.096103 15008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:06.096259 15008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5fdb40955d15407781df0d895678b2c4, State: Running, Role: LEADER
I20260927 18:35:06.096300 15003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:06.096459 15008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdb40955d15407781df0d895678b2c4" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 33721 } }
I20260927 18:35:06.097558 15010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fdb40955d15407781df0d895678b2c4. Latest consensus state: current_term: 1 leader_uuid: "5fdb40955d15407781df0d895678b2c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdb40955d15407781df0d895678b2c4" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 33721 } } }
I20260927 18:35:06.097570 15009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fdb40955d15407781df0d895678b2c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdb40955d15407781df0d895678b2c4" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 33721 } } }
I20260927 18:35:06.097653 15010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:06.097673 15009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4 [sys.catalog]: This master's current role is: LEADER
W20260927 18:35:06.098188 15023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:35:06.098577 15023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:35:06.098651 15024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:06.098835 15024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:06.100591 15024 catalog_manager.cc:1347] Generated new cluster ID: 2cca16707f8246c29031b81e6ed7710a
I20260927 18:35:06.100690 15024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:06.100924 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.094900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.5.151.254:33721)
0927 18:35:06.095155 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.095163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.095242 (+ 79us) server_negotiation.cc:407] Connection header received
0927 18:35:06.095383 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.095388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.095437 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.095566 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.095790 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.097596 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.098731 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.099336 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.099451 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:06.100129 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:06.100133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:06.100133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:06.100304 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:06.100336 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:06.100347 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:06.100419 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:06.100621 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 18:35:06.100706 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260927 18:35:06.107547 15024 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:06.108048 15024 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:06.113914 15024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fdb40955d15407781df0d895678b2c4: Generated new TSK 0
I20260927 18:35:06.114073 15024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:35:06.115836 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:33721
--builtin_ntp_servers=127.5.151.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 18:35:06.214412 15027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:06.214697 15027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:06.214735 15027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:06.215848 15027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:06.215909 15027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:06.215947 15027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:35:06.217229 15027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:06.217490 15027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:06.217784 15027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:35:06.222684 15027 server_base.cc:1034] running on GCE node
W20260927 18:35:06.222640 15033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.222640 15032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.222795 15035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.223002 15027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:06.223253 15027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:06.224408 15027 hybrid_clock.cc:648] HybridClock initialized: now 1790534106224396 us; error 28 us; skew 500 ppm
I20260927 18:35:06.225705 15027 webserver.cc:460] Webserver started at http://127.5.151.193:43075/ using document root <none> and password file <none>
I20260927 18:35:06.225946 15027 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:06.226011 15027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:06.226143 15027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:06.227280 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "cac222163432461683867c0d6cd9c1f6"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "a3e62db2b35406c82e7d1f35ad2a6d3f"
server_key_iv: "ac68485cb3c9f221d25088e6499447a7"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.227643 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "cac222163432461683867c0d6cd9c1f6"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "a3e62db2b35406c82e7d1f35ad2a6d3f"
server_key_iv: "ac68485cb3c9f221d25088e6499447a7"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.228935 15027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:35:06.229703 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.229887 15027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:35:06.229990 15027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "cac222163432461683867c0d6cd9c1f6"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "a3e62db2b35406c82e7d1f35ad2a6d3f"
server_key_iv: "ac68485cb3c9f221d25088e6499447a7"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.230058 15027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:06.248128 15027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:06.248427 15027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:06.248571 15027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:06.248782 15027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:06.249097 15027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:06.249150 15027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.249189 15027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:06.249215 15027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.254662 15027 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:35113
I20260927 18:35:06.254747 15154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:35113 every 8 connection(s)
I20260927 18:35:06.255055 15027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:06.260205 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.255761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:53445 (local address 127.5.151.254:33721)
0927 18:35:06.255827 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.255829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.256067 (+ 238us) server_negotiation.cc:407] Connection header received
0927 18:35:06.256605 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.256606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.256621 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.256677 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.257335 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.257853 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.258894 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.259112 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.259186 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:06.259940 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:06.259941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:06.259941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:06.259946 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:06.259960 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:06.259964 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:06.260010 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:06.260105 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 18:35:06.260131 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:35:06.260434 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.255762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:33721 (local address 127.5.151.193:53445)
0927 18:35:06.255949 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:06.255979 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:06.256493 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:06.256759 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:06.256762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:06.256770 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:06.257234 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:06.257239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.257947 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.257949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:06.258806 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:06.258809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.259193 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.259194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:06.259323 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:06.259806 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:35:06.259807 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:06.259833 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:35:06.259854 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:06.260008 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:06.260013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:06.260016 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:35:06.260094 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:35:06.260187 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260927 18:35:06.261122 15155 heartbeater.cc:347] Connected to a master server at 127.5.151.254:33721
I20260927 18:35:06.261210 15155 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:06.261185 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15027
I20260927 18:35:06.261263 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:35:06.261406 15155 heartbeater.cc:511] Master 127.5.151.254:33721 requested a full tablet report, sending...
I20260927 18:35:06.261485 5727 external_mini_cluster.cc:1428] Setting key 89cc0798997e2ce20457351f87004715
I20260927 18:35:06.261912 14967 ts_manager.cc:194] Registered new tserver with Master: cac222163432461683867c0d6cd9c1f6 (127.5.151.193:35113)
I20260927 18:35:06.262475 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:33721
--builtin_ntp_servers=127.5.151.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:35:06.262681 14967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.193:53445
W20260927 18:35:06.338339 15158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:06.338526 15158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:06.338586 15158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:06.339699 15158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:06.339740 15158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:06.339789 15158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:35:06.341051 15158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:06.341280 15158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:06.341552 15158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:06.345923 15164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.345930 15166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.345978 15163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.346151 15158 server_base.cc:1034] running on GCE node
I20260927 18:35:06.346406 15158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:06.346660 15158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:06.347810 15158 hybrid_clock.cc:648] HybridClock initialized: now 1790534106347800 us; error 26 us; skew 500 ppm
I20260927 18:35:06.349022 15158 webserver.cc:460] Webserver started at http://127.5.151.194:40589/ using document root <none> and password file <none>
I20260927 18:35:06.349251 15158 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:06.349295 15158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:06.349428 15158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:06.350376 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "be8f343b3ba047a9bb320f213555179a"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "4d99b2b88fe4ee3d0bd220426a2396fe"
server_key_iv: "cc045f07096bc39d5305c66f3c31f10b"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.350716 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "be8f343b3ba047a9bb320f213555179a"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "4d99b2b88fe4ee3d0bd220426a2396fe"
server_key_iv: "cc045f07096bc39d5305c66f3c31f10b"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.351881 15158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:35:06.352561 15172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.352730 15158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.352823 15158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "be8f343b3ba047a9bb320f213555179a"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "4d99b2b88fe4ee3d0bd220426a2396fe"
server_key_iv: "cc045f07096bc39d5305c66f3c31f10b"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.352897 15158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:06.382089 15158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:06.382371 15158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:06.382506 15158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:06.382731 15158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:06.383031 15158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:06.383078 15158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.383116 15158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:06.383139 15158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.388357 15158 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:39103
I20260927 18:35:06.388397 15285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:39103 every 8 connection(s)
I20260927 18:35:06.388690 15158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:06.393177 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.389359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:53975 (local address 127.5.151.254:33721)
0927 18:35:06.389443 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.389445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.389630 (+ 185us) server_negotiation.cc:407] Connection header received
0927 18:35:06.390077 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.390078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.390088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.390141 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.390744 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.391203 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.392047 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.392215 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.392260 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:06.392964 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:06.392965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:06.392966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:06.392970 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:06.392982 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:06.392986 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:06.393027 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:06.393104 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 18:35:06.393123 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:35:06.393416 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.389321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:33721 (local address 127.5.151.194:53975)
0927 18:35:06.389521 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:06.389553 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:06.389999 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:06.390189 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:06.390192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:06.390200 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:06.390681 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:06.390685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.391276 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.391276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:06.391985 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:06.391986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.392277 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.392277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:06.392375 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:06.392828 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:35:06.392828 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:06.392854 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:35:06.392873 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:06.393038 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:06.393042 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:06.393044 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:35:06.393094 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 18:35:06.393166 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260927 18:35:06.393908 15286 heartbeater.cc:347] Connected to a master server at 127.5.151.254:33721
I20260927 18:35:06.394007 15286 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:06.394176 15286 heartbeater.cc:511] Master 127.5.151.254:33721 requested a full tablet report, sending...
I20260927 18:35:06.394529 14967 ts_manager.cc:194] Registered new tserver with Master: be8f343b3ba047a9bb320f213555179a (127.5.151.194:39103)
I20260927 18:35:06.394919 14967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.194:53975
I20260927 18:35:06.396960 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15158
I20260927 18:35:06.397048 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:35:06.397245 5727 external_mini_cluster.cc:1428] Setting key 67b39892a5cec41721f80a684009bcd4
I20260927 18:35:06.398022 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:33721
--builtin_ntp_servers=127.5.151.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 18:35:06.473623 15289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:06.473798 15289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:06.473815 15289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:06.474963 15289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:06.474999 15289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:06.475023 15289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:35:06.476277 15289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:06.476496 15289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:06.476766 15289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:06.481326 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.481348 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.481377 15289 server_base.cc:1034] running on GCE node
W20260927 18:35:06.481349 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.481717 15289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:06.481969 15289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:06.483130 15289 hybrid_clock.cc:648] HybridClock initialized: now 1790534106483115 us; error 26 us; skew 500 ppm
I20260927 18:35:06.484236 15289 webserver.cc:460] Webserver started at http://127.5.151.195:34547/ using document root <none> and password file <none>
I20260927 18:35:06.484459 15289 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:06.484521 15289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:06.484658 15289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:06.485610 15289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "786187ff812a43fe9e07d80ecd0d8951"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "3464326aab54b4fed783b8c48b09512b"
server_key_iv: "0b8eddb066ca30ff34960842dbec0b6c"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.485960 15289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "786187ff812a43fe9e07d80ecd0d8951"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "3464326aab54b4fed783b8c48b09512b"
server_key_iv: "0b8eddb066ca30ff34960842dbec0b6c"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.487350 15289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:35:06.488088 15303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.488253 15289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:35:06.488345 15289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "786187ff812a43fe9e07d80ecd0d8951"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "3464326aab54b4fed783b8c48b09512b"
server_key_iv: "0b8eddb066ca30ff34960842dbec0b6c"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.488425 15289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:06.513927 15289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:06.514268 15289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:06.514451 15289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:06.514726 15289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:06.515074 15289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:06.515127 15289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.515161 15289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:06.515209 15289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.520598 15289 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:36343
I20260927 18:35:06.520673 15416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:36343 every 8 connection(s)
I20260927 18:35:06.521004 15289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:06.522306 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15289
I20260927 18:35:06.522411 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:35:06.522691 5727 external_mini_cluster.cc:1428] Setting key 1e4e1840817e9ed4fda992eea1237b01
I20260927 18:35:06.526325 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.521741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:37403 (local address 127.5.151.254:33721)
0927 18:35:06.521802 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.521804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.522159 (+ 355us) server_negotiation.cc:407] Connection header received
0927 18:35:06.522840 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.522841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.522852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.522923 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.523725 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.524235 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.525141 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.525319 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.525366 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:06.526109 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:06.526110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:06.526110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:06.526114 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:06.526127 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:06.526131 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:06.526169 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:06.526248 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 18:35:06.526272 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:35:06.526504 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.521739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:33721 (local address 127.5.151.195:37403)
0927 18:35:06.521926 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:06.521959 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:06.522727 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:06.522984 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:06.522988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:06.522996 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:06.523655 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:06.523660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.524315 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.524315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:06.525072 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:06.525073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:06.525391 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:06.525391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:06.525497 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:06.525989 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:35:06.525989 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:06.526016 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:35:06.526037 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:06.526175 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:06.526177 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:06.526180 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:35:06.526243 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:35:06.526311 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260927 18:35:06.526968 15417 heartbeater.cc:347] Connected to a master server at 127.5.151.254:33721
I20260927 18:35:06.527051 15417 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:06.527180 15417 heartbeater.cc:511] Master 127.5.151.254:33721 requested a full tablet report, sending...
I20260927 18:35:06.527498 14967 ts_manager.cc:194] Registered new tserver with Master: 786187ff812a43fe9e07d80ecd0d8951 (127.5.151.195:36343)
I20260927 18:35:06.527832 14967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.195:37403
I20260927 18:35:06.534693 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:35:06.536388 15007 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:06.535960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.5.151.254:33721)
0927 18:35:06.536016 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.536018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.536126 (+ 108us) server_negotiation.cc:407] Connection header received
0927 18:35:06.536160 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.536160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.536170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.536214 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.536319 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58822: BlockingRecv error: recv got EOF from 127.0.0.1:58822 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:35:06.536475 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15027
I20260927 18:35:06.540833 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15158
I20260927 18:35:06.544770 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15289
I20260927 18:35:06.548506 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 14936
2026-09-27T18:35:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (593 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T18:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:06Z Disabled control of system clock
I20260927 18:35:06.573541 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35967
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:35967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:35:06.650189 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:06.650380 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:06.650400 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:06.651464 15428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:06.651499 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:06.651511 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:06.651523 15428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:06.651535 15428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:06.652787 15428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:35967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:06.652966 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:06.653175 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:35:06.657689 15428 server_base.cc:1034] running on GCE node
W20260927 18:35:06.657661 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.657675 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.657655 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.658093 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:06.658370 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:06.659529 15428 hybrid_clock.cc:648] HybridClock initialized: now 1790534106659516 us; error 26 us; skew 500 ppm
I20260927 18:35:06.660594 15428 webserver.cc:460] Webserver started at http://127.5.151.254:37923/ using document root <none> and password file <none>
I20260927 18:35:06.660815 15428 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:06.660880 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:06.661029 15428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:06.661986 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "a855a76f3cb049618deed321d40eda47"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "e8821857c32084d103266bacf07d86ab"
server_key_iv: "2caadbf810074e5f6728120e34017394"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.662338 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "a855a76f3cb049618deed321d40eda47"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "e8821857c32084d103266bacf07d86ab"
server_key_iv: "2caadbf810074e5f6728120e34017394"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.663599 15428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:35:06.664245 15442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.664415 15428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:35:06.664507 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "a855a76f3cb049618deed321d40eda47"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "e8821857c32084d103266bacf07d86ab"
server_key_iv: "2caadbf810074e5f6728120e34017394"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.664592 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:06.667752 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:06.667999 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:06.668131 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:06.671818 15428 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:35967
I20260927 18:35:06.671845 15494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:35967 every 8 connection(s)
I20260927 18:35:06.672148 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:06.672715 15495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:06.675194 15495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47: Bootstrap starting.
I20260927 18:35:06.675668 15495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:06.675895 15495 log.cc:824] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:06.676467 15495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47: No bootstrap required, opened a new log
I20260927 18:35:06.677201 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15428
I20260927 18:35:06.677295 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:35:06.677484 5727 external_mini_cluster.cc:1428] Setting key c2a8327de90aaefb290c4186da57ac81
I20260927 18:35:06.678442 15495 raft_consensus.cc:348] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a855a76f3cb049618deed321d40eda47" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35967 } }
I20260927 18:35:06.678622 15495 raft_consensus.cc:374] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:06.678696 15495 raft_consensus.cc:729] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a855a76f3cb049618deed321d40eda47, State: Initialized, Role: FOLLOWER
I20260927 18:35:06.678884 15495 consensus_queue.cc:260] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a855a76f3cb049618deed321d40eda47" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35967 } }
I20260927 18:35:06.678980 15495 raft_consensus.cc:388] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:06.679011 15495 raft_consensus.cc:482] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:06.679054 15495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:06.679718 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.677996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.5.151.254:35967)
0927 18:35:06.678219 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.678228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.678261 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:35:06.678401 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.678408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.678451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.678622 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.679291 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:06.679293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:06.679294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:06.679297 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:06.679330 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:06.679341 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:06.679394 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:06.679478 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:35:06.679541 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260927 18:35:06.679708 15495 raft_consensus.cc:504] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a855a76f3cb049618deed321d40eda47" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35967 } }
I20260927 18:35:06.679862 15495 leader_election.cc:302] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a855a76f3cb049618deed321d40eda47; no voters:
I20260927 18:35:06.680006 15495 leader_election.cc:288] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:06.680258 15500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:06.680266 15495 sys_catalog.cc:564] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:06.680433 15500 raft_consensus.cc:686] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [term 1 LEADER]: Becoming Leader. State: Replica: a855a76f3cb049618deed321d40eda47, State: Running, Role: LEADER
I20260927 18:35:06.680519 15500 consensus_queue.cc:237] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a855a76f3cb049618deed321d40eda47" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35967 } }
I20260927 18:35:06.681641 15502 sys_catalog.cc:455] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a855a76f3cb049618deed321d40eda47. Latest consensus state: current_term: 1 leader_uuid: "a855a76f3cb049618deed321d40eda47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a855a76f3cb049618deed321d40eda47" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35967 } } }
I20260927 18:35:06.681782 15501 sys_catalog.cc:455] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a855a76f3cb049618deed321d40eda47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a855a76f3cb049618deed321d40eda47" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35967 } } }
I20260927 18:35:06.681931 15502 sys_catalog.cc:458] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:06.681938 15501 sys_catalog.cc:458] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47 [sys.catalog]: This master's current role is: LEADER
W20260927 18:35:06.682112 15515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:35:06.682156 15515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:35:06.682204 15517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:06.682335 15517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:06.683655 15517 catalog_manager.cc:1347] Generated new cluster ID: 7219d37d1ed8496e9979ac6ca1fcbd84
I20260927 18:35:06.683704 15517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:06.694615 15517 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:06.694756 15517 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:06.700126 15517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a855a76f3cb049618deed321d40eda47: Generated new TSK 0
I20260927 18:35:06.700289 15517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:35:06.707744 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35967
--builtin_ntp_servers=127.5.151.212:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:35:06.786193 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:06.786414 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:06.786459 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:06.787639 15519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:06.787700 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:06.787751 15519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:35:06.789065 15519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:06.789320 15519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:06.789605 15519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:06.795388 15527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.795387 15524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.795591 15519 server_base.cc:1034] running on GCE node
W20260927 18:35:06.795450 15525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.795923 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:06.796154 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:06.797310 15519 hybrid_clock.cc:648] HybridClock initialized: now 1790534106797297 us; error 28 us; skew 500 ppm
I20260927 18:35:06.798592 15519 webserver.cc:460] Webserver started at http://127.5.151.193:42695/ using document root <none> and password file <none>
I20260927 18:35:06.798846 15519 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:06.798913 15519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:06.799057 15519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:06.800065 15519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "146f633330f0409b805265e25c3ce720"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "8a8f4984039009b06da577db96613538"
server_key_iv: "7b8254ea8d77c85947b29af195f53a26"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.800419 15519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "146f633330f0409b805265e25c3ce720"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "8a8f4984039009b06da577db96613538"
server_key_iv: "7b8254ea8d77c85947b29af195f53a26"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.801674 15519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:35:06.802467 15533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.802661 15519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:35:06.802752 15519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "146f633330f0409b805265e25c3ce720"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "8a8f4984039009b06da577db96613538"
server_key_iv: "7b8254ea8d77c85947b29af195f53a26"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.802834 15519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:06.805912 15519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:06.806196 15519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:06.806319 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:06.806526 15519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:06.806849 15519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:06.806902 15519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.806941 15519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:06.806965 15519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.812515 15519 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:34981
I20260927 18:35:06.812546 15646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:34981 every 8 connection(s)
I20260927 18:35:06.812839 15519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:06.815433 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.813441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:56057 (local address 127.5.151.254:35967)
0927 18:35:06.813513 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.813516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.813722 (+ 206us) server_negotiation.cc:407] Connection header received
0927 18:35:06.814106 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.814107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.814112 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.814160 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.815259 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:06.815259 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:06.815260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:06.815263 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:06.815274 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:06.815277 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:06.815319 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:06.815372 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 18:35:06.815383 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:35:06.815593 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.813427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35967 (local address 127.5.151.193:56057)
0927 18:35:06.813610 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:06.813640 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:06.814034 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:06.814206 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:06.814209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:06.814216 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:06.815141 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:35:06.815141 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:06.815166 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:35:06.815192 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:06.815317 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:06.815319 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:06.815321 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:35:06.815369 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:35:06.815411 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260927 18:35:06.816049 15647 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35967
I20260927 18:35:06.816143 15647 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:06.816278 15647 heartbeater.cc:511] Master 127.5.151.254:35967 requested a full tablet report, sending...
I20260927 18:35:06.816726 15459 ts_manager.cc:194] Registered new tserver with Master: 146f633330f0409b805265e25c3ce720 (127.5.151.193:34981)
I20260927 18:35:06.822450 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15519
I20260927 18:35:06.822557 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:35:06.822781 5727 external_mini_cluster.cc:1428] Setting key a0a563ae29ba239a478f5df1bc4b1f12
I20260927 18:35:06.823619 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35967
--builtin_ntp_servers=127.5.151.212:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:35:06.899894 15650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:06.900071 15650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:06.900110 15650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:06.901252 15650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:06.901289 15650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:06.901314 15650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:35:06.902730 15650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:06.902957 15650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:06.903234 15650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:06.908022 15656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.908113 15650 server_base.cc:1034] running on GCE node
W20260927 18:35:06.908038 15658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:06.908059 15655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:06.908435 15650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:06.908684 15650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:06.909816 15650 hybrid_clock.cc:648] HybridClock initialized: now 1790534106909812 us; error 34 us; skew 500 ppm
I20260927 18:35:06.914661 15650 webserver.cc:460] Webserver started at http://127.5.151.194:35837/ using document root <none> and password file <none>
I20260927 18:35:06.914914 15650 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:06.914968 15650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:06.915135 15650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:06.916154 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "dd82c24c9f754df890b3699069f9329f"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "2410a00be298dabb43d823f9dc96cbf9"
server_key_iv: "4a5ab51048a1a104c2fec6cc57075749"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.916500 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "dd82c24c9f754df890b3699069f9329f"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "2410a00be298dabb43d823f9dc96cbf9"
server_key_iv: "4a5ab51048a1a104c2fec6cc57075749"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.917747 15650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:35:06.918399 15664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.918601 15650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:35:06.918689 15650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "dd82c24c9f754df890b3699069f9329f"
format_stamp: "Formatted at 2026-09-27 18:35:06 on dist-test-slave-19nl"
server_key: "2410a00be298dabb43d823f9dc96cbf9"
server_key_iv: "4a5ab51048a1a104c2fec6cc57075749"
server_key_version: "encryptionkey@0"
I20260927 18:35:06.918757 15650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:06.921849 15650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:06.922041 15650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:06.922127 15650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:06.922268 15650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:06.922492 15650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:06.922520 15650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.922539 15650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:06.922611 15650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:06.928154 15650 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:42983
I20260927 18:35:06.928198 15777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:42983 every 8 connection(s)
I20260927 18:35:06.928479 15650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:06.931085 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.929122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:60383 (local address 127.5.151.254:35967)
0927 18:35:06.929185 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:06.929187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:06.929370 (+ 183us) server_negotiation.cc:407] Connection header received
0927 18:35:06.929774 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:06.929775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:06.929779 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:06.929833 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:06.930874 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:06.930876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:06.930876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:06.930880 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:06.930898 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:06.930902 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:06.930951 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:06.931023 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:35:06.931035 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:35:06.931267 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:06.929086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35967 (local address 127.5.151.194:60383)
0927 18:35:06.929249 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:06.929286 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:06.929689 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:06.929878 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:06.929881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:06.929888 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:06.930747 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:35:06.930747 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:06.930773 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:35:06.930799 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:06.930959 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:06.930963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:06.930965 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:35:06.931020 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 18:35:06.931064 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260927 18:35:06.931811 15778 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35967
I20260927 18:35:06.931910 15778 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:06.932045 15778 heartbeater.cc:511] Master 127.5.151.254:35967 requested a full tablet report, sending...
I20260927 18:35:06.932403 15459 ts_manager.cc:194] Registered new tserver with Master: dd82c24c9f754df890b3699069f9329f (127.5.151.194:42983)
I20260927 18:35:06.937780 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15650
I20260927 18:35:06.937870 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:35:06.938006 5727 external_mini_cluster.cc:1428] Setting key 0e3a8a21c8b2f09169f209d3f6bce1d3
I20260927 18:35:06.938952 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35967
--builtin_ntp_servers=127.5.151.212:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:35:07.015952 15781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:07.016171 15781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:07.016206 15781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:07.017300 15781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:07.017360 15781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:07.017401 15781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:35:07.018718 15781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:07.018963 15781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:07.019239 15781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:07.023949 15787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:07.023952 15789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:07.024123 15781 server_base.cc:1034] running on GCE node
W20260927 18:35:07.024209 15786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:07.024402 15781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:07.024647 15781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:07.025806 15781 hybrid_clock.cc:648] HybridClock initialized: now 1790534107025788 us; error 30 us; skew 500 ppm
I20260927 18:35:07.027011 15781 webserver.cc:460] Webserver started at http://127.5.151.195:36785/ using document root <none> and password file <none>
I20260927 18:35:07.027248 15781 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:07.027314 15781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:07.027444 15781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:07.028437 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "b7fd72cf698d476c9d3c3f4acdd7ffaa"
format_stamp: "Formatted at 2026-09-27 18:35:07 on dist-test-slave-19nl"
server_key: "cac0486b6763388cc14228d584172531"
server_key_iv: "e3544868345b1f08de3037afa2b3acb6"
server_key_version: "encryptionkey@0"
I20260927 18:35:07.028792 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "b7fd72cf698d476c9d3c3f4acdd7ffaa"
format_stamp: "Formatted at 2026-09-27 18:35:07 on dist-test-slave-19nl"
server_key: "cac0486b6763388cc14228d584172531"
server_key_iv: "e3544868345b1f08de3037afa2b3acb6"
server_key_version: "encryptionkey@0"
I20260927 18:35:07.029986 15781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:35:07.030764 15795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:07.030921 15781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 18:35:07.031034 15781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "b7fd72cf698d476c9d3c3f4acdd7ffaa"
format_stamp: "Formatted at 2026-09-27 18:35:07 on dist-test-slave-19nl"
server_key: "cac0486b6763388cc14228d584172531"
server_key_iv: "e3544868345b1f08de3037afa2b3acb6"
server_key_version: "encryptionkey@0"
I20260927 18:35:07.031104 15781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:07.034307 15781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:07.034534 15781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:07.034695 15781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:07.034889 15781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:07.035241 15781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:07.035295 15781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:07.035333 15781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:07.035359 15781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:07.040877 15781 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:32885
I20260927 18:35:07.040922 15908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:32885 every 8 connection(s)
I20260927 18:35:07.041204 15781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:07.042843 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15781
I20260927 18:35:07.042912 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:35:07.043081 5727 external_mini_cluster.cc:1428] Setting key e0ea62414d4912a6eb6802ffae3d0f1b
I20260927 18:35:07.044046 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:07.041805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:57147 (local address 127.5.151.254:35967)
0927 18:35:07.041889 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:07.041891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:07.042058 (+ 167us) server_negotiation.cc:407] Connection header received
0927 18:35:07.042452 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:07.042453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:07.042459 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:07.042508 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:07.043821 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:07.043823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:07.043823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:35:07.043827 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:07.043840 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:35:07.043845 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:07.043897 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:07.043978 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 18:35:07.043998 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:35:07.044242 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:07.041777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35967 (local address 127.5.151.195:57147)
0927 18:35:07.041950 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:07.041978 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:07.042374 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:07.042598 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:07.042602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:07.042612 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:07.043664 (+ 1052us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:35:07.043665 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:07.043704 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:35:07.043728 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:07.043895 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:07.043900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:07.043902 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:35:07.043994 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 18:35:07.044039 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260927 18:35:07.044660 15909 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35967
I20260927 18:35:07.044729 15909 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:07.044881 15909 heartbeater.cc:511] Master 127.5.151.254:35967 requested a full tablet report, sending...
I20260927 18:35:07.045215 15459 ts_manager.cc:194] Registered new tserver with Master: b7fd72cf698d476c9d3c3f4acdd7ffaa (127.5.151.195:32885)
I20260927 18:35:07.054719 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:35:07.056456 15499 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:35:07.055949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.5.151.254:35967)
0927 18:35:07.056013 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:07.056015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:07.056117 (+ 102us) server_negotiation.cc:407] Connection header received
0927 18:35:07.056198 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:07.056199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:07.056204 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:07.056252 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:07.056396 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47426: BlockingRecv error: recv got EOF from 127.0.0.1:47426 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:35:07.056493 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15519
I20260927 18:35:07.060624 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15650
I20260927 18:35:07.064625 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15781
I20260927 18:35:07.068377 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15428
2026-09-27T18:35:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (519 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:35:07 dist-test-slave-19nl krb5kdc[15919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:35:07 dist-test-slave-19nl krb5kdc[15919](info): set up 2 sockets
Sep 27 18:35:07 dist-test-slave-19nl krb5kdc[15919](info): commencing operation
krb5kdc: starting...
W20260927 18:35:08.063556 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:35:08.171499 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34713
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:34713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:08.248143 15932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:08.248343 15932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:08.248365 15932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:08.249495 15932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:08.249534 15932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:08.249548 15932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:08.249598 15932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:08.249619 15932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:08.251092 15932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:34713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:08.251312 15932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:08.251534 15932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:08.256038 15938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:08.256088 15937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:08.256247 15932 server_base.cc:1034] running on GCE node
W20260927 18:35:08.256045 15940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:08.256558 15932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:08.256858 15932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:08.258020 15932 hybrid_clock.cc:648] HybridClock initialized: now 1790534108258012 us; error 38 us; skew 500 ppm
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:08.260418 15932 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:35:08.260833 15932 webserver.cc:460] Webserver started at http://127.5.151.254:38999/ using document root <none> and password file <none>
I20260927 18:35:08.261061 15932 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:08.261126 15932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:08.261273 15932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:08.262284 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "412b75cc259840d197f3d5a8ba743a43"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "ee68b68ae1e935bfd273d9fd4ff0739c"
server_key_iv: "bf25dfce349b8bae246049fe0ba6c452"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.262691 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "412b75cc259840d197f3d5a8ba743a43"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "ee68b68ae1e935bfd273d9fd4ff0739c"
server_key_iv: "bf25dfce349b8bae246049fe0ba6c452"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.263939 15932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:35:08.264632 15947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.264791 15932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:35:08.264885 15932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "412b75cc259840d197f3d5a8ba743a43"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "ee68b68ae1e935bfd273d9fd4ff0739c"
server_key_iv: "bf25dfce349b8bae246049fe0ba6c452"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.264967 15932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:08.285586 15932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:08.287498 15932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:08.287658 15932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:08.291505 15932 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:34713
I20260927 18:35:08.291544 15999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:34713 every 8 connection(s)
I20260927 18:35:08.291906 15932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:08.292837 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:08.294945 16000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43: Bootstrap starting.
I20260927 18:35:08.295454 16000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:08.295672 16000 log.cc:824] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:08.295794 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 15932
I20260927 18:35:08.295899 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:35:08.296066 5727 external_mini_cluster.cc:1428] Setting key c4429ca0cbc31f95f859f3d765da59b6
I20260927 18:35:08.296394 16000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43: No bootstrap required, opened a new log
I20260927 18:35:08.298043 16000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b75cc259840d197f3d5a8ba743a43" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34713 } }
I20260927 18:35:08.298198 16000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:08.298244 16000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412b75cc259840d197f3d5a8ba743a43, State: Initialized, Role: FOLLOWER
I20260927 18:35:08.298393 16000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b75cc259840d197f3d5a8ba743a43" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34713 } }
I20260927 18:35:08.298481 16000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:08.298517 16000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:08.298593 16000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:08.299599 16000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b75cc259840d197f3d5a8ba743a43" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34713 } }
I20260927 18:35:08.299682 16000 leader_election.cc:302] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412b75cc259840d197f3d5a8ba743a43; no voters:
I20260927 18:35:08.299863 16000 leader_election.cc:288] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:08.299911 16005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:08.300103 16000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:08.300104 16005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [term 1 LEADER]: Becoming Leader. State: Replica: 412b75cc259840d197f3d5a8ba743a43, State: Running, Role: LEADER
I20260927 18:35:08.300328 16005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b75cc259840d197f3d5a8ba743a43" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34713 } }
I20260927 18:35:08.300694 16006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412b75cc259840d197f3d5a8ba743a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b75cc259840d197f3d5a8ba743a43" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34713 } } }
I20260927 18:35:08.300776 16006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:08.300915 16007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412b75cc259840d197f3d5a8ba743a43. Latest consensus state: current_term: 1 leader_uuid: "412b75cc259840d197f3d5a8ba743a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b75cc259840d197f3d5a8ba743a43" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34713 } } }
I20260927 18:35:08.300953 16007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:08.301285 16014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:08.301800 16014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:08.303493 16014 catalog_manager.cc:1347] Generated new cluster ID: c1a887c79dac4172860122184f8c2f77
I20260927 18:35:08.303541 16014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:08.308318 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.296523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.5.151.254:34713)
0927 18:35:08.296666 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.296675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.296707 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:35:08.298033 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.298042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.298092 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.298207 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:08.299123 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.300231 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.301348 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.301925 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.302021 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.304917 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:08.304921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:08.304922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:08.304932 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:08.306727 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.307117 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.307119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.307120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.307244 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.307456 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.307457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.307457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.307720 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:08.307873 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.308015 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.308142 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260927 18:35:08.310390 16014 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:08.310920 16014 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:08.313171 16014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43: Generated new TSK 0
I20260927 18:35:08.313308 16014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:35:08.335650 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34713
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:08.410956 16026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:08.411142 16026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:08.411177 16026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:08.412238 16026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:08.412298 16026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:08.412340 16026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:35:08.413697 16026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:08.413925 16026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:08.414115 16026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:08.418694 16034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:08.418776 16026 server_base.cc:1034] running on GCE node
W20260927 18:35:08.418876 16031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:08.418736 16032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:08.419106 16026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:08.419341 16026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:08.420490 16026 hybrid_clock.cc:648] HybridClock initialized: now 1790534108420478 us; error 26 us; skew 500 ppm
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:08.423278 16026 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:35:08.423714 16026 webserver.cc:460] Webserver started at http://127.5.151.193:40369/ using document root <none> and password file <none>
I20260927 18:35:08.423946 16026 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:08.424013 16026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:08.424156 16026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:08.425137 16026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "73242aa9c576446bb5fd27d69cca7af2"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "a4ef3a3f9f37d495cd2f55d44d39fe17"
server_key_iv: "75701e78ccfecb51715395881213b7b8"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.425482 16026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "73242aa9c576446bb5fd27d69cca7af2"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "a4ef3a3f9f37d495cd2f55d44d39fe17"
server_key_iv: "75701e78ccfecb51715395881213b7b8"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.426795 16026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 18:35:08.427456 16041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.427623 16026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:35:08.427707 16026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "73242aa9c576446bb5fd27d69cca7af2"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "a4ef3a3f9f37d495cd2f55d44d39fe17"
server_key_iv: "75701e78ccfecb51715395881213b7b8"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.427784 16026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:08.439386 16026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:08.441545 16026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:08.441699 16026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:08.441910 16026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:08.442190 16026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:08.442239 16026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.442282 16026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:08.442312 16026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.447769 16026 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:34329
I20260927 18:35:08.447825 16154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:34329 every 8 connection(s)
I20260927 18:35:08.448100 16026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:35:08.449869 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 16026
I20260927 18:35:08.449941 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:35:08.450127 5727 external_mini_cluster.cc:1428] Setting key 8ec51015b51dfebfe7057ffe6713d43d
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:08.457902 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.448733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34713 (local address 127.5.151.193:32769)
0927 18:35:08.448887 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.448920 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.449367 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.449555 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.449558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.450225 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:08.450501 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.450508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.451405 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.451405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.452240 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.452243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.452666 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.452667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.452792 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.453334 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:08.453343 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:08.454909 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:08.456688 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:08.456693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:08.456695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:08.456893 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:08.457103 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:08.457103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:08.457103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:08.457190 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:08.457463 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:08.457466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:08.457569 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.457645 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.457752 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260927 18:35:08.457896 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.448749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:32769 (local address 127.5.151.254:34713)
0927 18:35:08.448795 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.448797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.449005 (+ 208us) server_negotiation.cc:407] Connection header received
0927 18:35:08.449443 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.449444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.449456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.449505 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:08.450608 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.451315 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.452328 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.452534 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.452653 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.455004 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:08.455006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:08.455007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:08.455017 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:08.456602 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.456965 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.456966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.456966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.457030 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.457252 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.457252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.457253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.457396 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:08.457468 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.457694 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.457782 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
I20260927 18:35:08.458752 16155 heartbeater.cc:347] Connected to a master server at 127.5.151.254:34713
I20260927 18:35:08.458855 16155 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:08.459023 16155 heartbeater.cc:511] Master 127.5.151.254:34713 requested a full tablet report, sending...
I20260927 18:35:08.459506 15964 ts_manager.cc:194] Registered new tserver with Master: 73242aa9c576446bb5fd27d69cca7af2 (127.5.151.193:34329)
I20260927 18:35:08.460090 15964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:32769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:35:08.472944 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34713
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:08.550151 16160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:08.550318 16160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:08.550338 16160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:08.551365 16160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:08.551407 16160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:08.551430 16160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:35:08.552686 16160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:08.552865 16160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:08.553016 16160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:08.557560 16168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:08.557615 16165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:08.557664 16160 server_base.cc:1034] running on GCE node
W20260927 18:35:08.557646 16166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:08.557971 16160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:08.558220 16160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:08.559376 16160 hybrid_clock.cc:648] HybridClock initialized: now 1790534108559352 us; error 33 us; skew 500 ppm
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:08.561705 16160 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:35:08.562088 16160 webserver.cc:460] Webserver started at http://127.5.151.194:45033/ using document root <none> and password file <none>
I20260927 18:35:08.562330 16160 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:08.562395 16160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:08.562544 16160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:08.563484 16160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "8a2bc5d4cb9e4828b7ee20f64de31893"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "bc56428eaac60f914f25a609ce84898a"
server_key_iv: "68ab4c4132034f28f6d3be02fccce6ae"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.563814 16160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "8a2bc5d4cb9e4828b7ee20f64de31893"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "bc56428eaac60f914f25a609ce84898a"
server_key_iv: "68ab4c4132034f28f6d3be02fccce6ae"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.565038 16160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:35:08.565670 16175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.565819 16160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:35:08.565908 16160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "8a2bc5d4cb9e4828b7ee20f64de31893"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "bc56428eaac60f914f25a609ce84898a"
server_key_iv: "68ab4c4132034f28f6d3be02fccce6ae"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.565982 16160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:08.585290 16160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:08.587463 16160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:08.587615 16160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:08.587814 16160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:08.588099 16160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:08.588150 16160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.588181 16160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:08.588230 16160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.593701 16160 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:45955
I20260927 18:35:08.593732 16288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:45955 every 8 connection(s)
I20260927 18:35:08.594017 16160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:08.596843 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 16160
I20260927 18:35:08.596956 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:35:08.597157 5727 external_mini_cluster.cc:1428] Setting key 967c68a480ec25bb650f8c23e4aea3a0
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:08.603785 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.594781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:42271 (local address 127.5.151.254:34713)
0927 18:35:08.594845 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.594847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.595039 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:35:08.595474 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.595475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.595484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.595538 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:08.596353 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.596832 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.598196 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.598388 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.598443 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.600669 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:08.600671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:08.600671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:08.600679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:08.602365 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.602798 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.602800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.602800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.602867 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.603094 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.603095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.603095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.603250 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:08.603331 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.603566 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.603683 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:35:08.603873 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.594765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34713 (local address 127.5.151.194:42271)
0927 18:35:08.594913 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.594952 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.595396 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.595585 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.595588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.596096 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:08.596279 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.596283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.596920 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.596921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.598115 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.598119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.598467 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.598469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.598650 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.599105 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:08.599115 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:08.600580 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:08.602446 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:08.602452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:08.602454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:08.602727 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:08.602942 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:08.602942 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:08.602942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:08.603034 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:08.603331 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:08.603335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:08.603442 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.603517 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.603629 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260927 18:35:08.604821 16289 heartbeater.cc:347] Connected to a master server at 127.5.151.254:34713
I20260927 18:35:08.604962 16289 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:08.605161 16289 heartbeater.cc:511] Master 127.5.151.254:34713 requested a full tablet report, sending...
I20260927 18:35:08.605597 15964 ts_manager.cc:194] Registered new tserver with Master: 8a2bc5d4cb9e4828b7ee20f64de31893 (127.5.151.194:45955)
I20260927 18:35:08.606106 15964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:42271
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:35:08.622762 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34713
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:08.697748 16294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:08.697925 16294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:08.697947 16294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:08.699007 16294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:08.699043 16294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:08.699066 16294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:35:08.700291 16294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:34713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:08.700528 16294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:08.700686 16294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:08.705036 16300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:08.705077 16299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:08.705036 16302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:08.705283 16294 server_base.cc:1034] running on GCE node
I20260927 18:35:08.705485 16294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:08.705703 16294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:08.706843 16294 hybrid_clock.cc:648] HybridClock initialized: now 1790534108706827 us; error 32 us; skew 500 ppm
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:08.708930 16294 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:35:08.709357 16294 webserver.cc:460] Webserver started at http://127.5.151.195:46755/ using document root <none> and password file <none>
I20260927 18:35:08.709576 16294 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:08.709640 16294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:08.709786 16294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:35:08.710793 16294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "4ef3c0b74746472e8058bfbdc8bb08a7"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "057c2b993c5c8992be0a227399c31428"
server_key_iv: "9b7faa852d4ae70dbdad570a326d5798"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.711112 16294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "4ef3c0b74746472e8058bfbdc8bb08a7"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "057c2b993c5c8992be0a227399c31428"
server_key_iv: "9b7faa852d4ae70dbdad570a326d5798"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.712369 16294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:35:08.713025 16309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.713179 16294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:35:08.713270 16294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "4ef3c0b74746472e8058bfbdc8bb08a7"
format_stamp: "Formatted at 2026-09-27 18:35:08 on dist-test-slave-19nl"
server_key: "057c2b993c5c8992be0a227399c31428"
server_key_iv: "9b7faa852d4ae70dbdad570a326d5798"
server_key_version: "encryptionkey@0"
I20260927 18:35:08.713346 16294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:08.725764 16294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:08.727394 16294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:08.727595 16294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:08.727806 16294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:08.728077 16294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:08.728129 16294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.728166 16294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:08.728195 16294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:08.733819 16294 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:38565
I20260927 18:35:08.733853 16422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:38565 every 8 connection(s)
I20260927 18:35:08.734150 16294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:08.736945 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 16294
I20260927 18:35:08.737061 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:35:08.737258 5727 external_mini_cluster.cc:1428] Setting key 2f5601b31676a3b894200859b3e93e02
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:08.742933 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.734927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:42351 (local address 127.5.151.254:34713)
0927 18:35:08.735001 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.735003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.735176 (+ 173us) server_negotiation.cc:407] Connection header received
0927 18:35:08.735604 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.735605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.735615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.735662 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:08.736452 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.736919 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.738242 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.738457 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.738538 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.740499 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:08.740500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:08.740500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:08.740508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:08.741614 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.742002 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.742003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.742004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.742063 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.742299 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.742300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.742300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.742411 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:08.742485 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.742771 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.742845 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:35:08.743099 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.734891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:34713 (local address 127.5.151.195:42351)
0927 18:35:08.735064 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.735095 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.735510 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.735709 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.735712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.736216 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:08.736382 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.736385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.737011 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.737013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.738162 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.738165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.738565 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.738567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.738698 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.739170 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:08.739179 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:08.740409 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:08.741693 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:08.741699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:08.741701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:08.741919 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:08.742131 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:08.742132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:08.742132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:08.742231 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:08.742476 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:08.742481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:08.742645 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.742749 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.742895 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260927 18:35:08.743517 16423 heartbeater.cc:347] Connected to a master server at 127.5.151.254:34713
I20260927 18:35:08.743600 16423 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:08.743719 16423 heartbeater.cc:511] Master 127.5.151.254:34713 requested a full tablet report, sending...
I20260927 18:35:08.744027 15964 ts_manager.cc:194] Registered new tserver with Master: 4ef3c0b74746472e8058bfbdc8bb08a7 (127.5.151.195:38565)
I20260927 18:35:08.744367 15964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:42351
I20260927 18:35:08.748869 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:35:08.755313 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.750019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.5.151.254:34713)
0927 18:35:08.750052 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.750054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.750177 (+ 123us) server_negotiation.cc:407] Connection header received
0927 18:35:08.750221 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.750222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.750229 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.750271 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:08.750826 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.751233 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.751879 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.752036 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.752080 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.753155 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:08.753156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:08.753156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:08.753160 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:08.754187 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.754519 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.754520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.754520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.754593 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:08.754793 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:08.754794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:08.754794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:08.754927 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:08.755023 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.755148 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.755216 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:35:08.756834 15964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:35:08.758246 15964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:35:08.764087 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.760117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:38565 (local address 127.0.0.1:41144)
0927 18:35:08.760463 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.760496 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.760575 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.760860 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.760862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.760864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:08.760921 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.760922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.761856 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.761857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.763004 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.763007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.763786 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.763787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.763865 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.763885 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.763971 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.764019 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":72,"threads_started":1}
I20260927 18:35:08.764094 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.760187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.5.151.195:38565)
0927 18:35:08.760346 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.760351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.760650 (+ 299us) server_negotiation.cc:407] Connection header received
0927 18:35:08.760687 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.760689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.760701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.760766 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:08.760984 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.761726 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.763094 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.763671 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.763787 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.763841 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.763987 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.764023 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260927 18:35:08.764607 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.760208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.5.151.194:45955)
0927 18:35:08.760396 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.760400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.760552 (+ 152us) server_negotiation.cc:407] Connection header received
0927 18:35:08.760692 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.760694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.760711 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.760761 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:08.761140 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.762247 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.763480 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.764133 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.764200 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.764259 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.764470 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.764509 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260927 18:35:08.764817 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.760092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:45955 (local address 127.0.0.1:58230)
0927 18:35:08.760449 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.760490 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.760582 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.761025 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.761026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.761028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:08.761061 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.761062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.762389 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.762390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.763386 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.763390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.764322 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.764323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.764386 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.764389 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.764695 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.764743 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":51,"threads_started":1}
I20260927 18:35:08.764824 16357 tablet_service.cc:1467] Processing CreateTablet for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 (DEFAULT_TABLE table=test-table [id=4889e6a783a840bea24bcbf1d5b6111b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:08.765163 16357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:08.765381 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.760325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:34329 (local address 127.0.0.1:54728)
0927 18:35:08.760608 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.760617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.760674 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.760861 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.760863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.760866 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:08.760924 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.760926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.762263 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.762264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.763326 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.763329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.764802 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.764803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.765061 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.765066 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.765202 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.765309 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260927 18:35:08.765524 16223 tablet_service.cc:1467] Processing CreateTablet for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 (DEFAULT_TABLE table=test-table [id=4889e6a783a840bea24bcbf1d5b6111b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:08.765719 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.760442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.5.151.193:34329)
0927 18:35:08.760609 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.760614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.760667 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:35:08.760747 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.760748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.760762 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.760836 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:08.761008 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.762158 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.763406 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.764506 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.765488 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.765551 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.765576 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.765612 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260927 18:35:08.765825 16223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:08.768354 16437 tablet_bootstrap.cc:492] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Bootstrap starting.
I20260927 18:35:08.768949 16438 tablet_bootstrap.cc:492] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Bootstrap starting.
I20260927 18:35:08.769065 16437 tablet_bootstrap.cc:654] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:08.769519 16438 tablet_bootstrap.cc:654] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:08.769701 16437 log.cc:824] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:08.769704 16089 tablet_service.cc:1467] Processing CreateTablet for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 (DEFAULT_TABLE table=test-table [id=4889e6a783a840bea24bcbf1d5b6111b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:08.770020 16089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:08.770064 16438 log.cc:824] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:08.770458 16437 tablet_bootstrap.cc:492] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: No bootstrap required, opened a new log
I20260927 18:35:08.770529 16437 ts_tablet_manager.cc:1397] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:35:08.770797 16438 tablet_bootstrap.cc:492] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: No bootstrap required, opened a new log
I20260927 18:35:08.770874 16438 ts_tablet_manager.cc:1397] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:35:08.772091 16438 raft_consensus.cc:348] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.772244 16438 raft_consensus.cc:374] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:08.772292 16438 raft_consensus.cc:729] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2bc5d4cb9e4828b7ee20f64de31893, State: Initialized, Role: FOLLOWER
I20260927 18:35:08.772400 16438 consensus_queue.cc:260] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.772411 16437 raft_consensus.cc:348] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.772568 16437 raft_consensus.cc:374] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:08.772615 16437 raft_consensus.cc:729] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef3c0b74746472e8058bfbdc8bb08a7, State: Initialized, Role: FOLLOWER
I20260927 18:35:08.772656 16438 ts_tablet_manager.cc:1428] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:35:08.772724 16437 consensus_queue.cc:260] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.772768 16289 heartbeater.cc:503] Master 127.5.151.254:34713 was elected leader, sending a full tablet report...
I20260927 18:35:08.773006 16437 ts_tablet_manager.cc:1428] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:35:08.773113 16423 heartbeater.cc:503] Master 127.5.151.254:34713 was elected leader, sending a full tablet report...
I20260927 18:35:08.773847 16443 tablet_bootstrap.cc:492] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2: Bootstrap starting.
I20260927 18:35:08.774372 16443 tablet_bootstrap.cc:654] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:08.774960 16443 log.cc:824] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:08.775749 16443 tablet_bootstrap.cc:492] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2: No bootstrap required, opened a new log
I20260927 18:35:08.775815 16443 ts_tablet_manager.cc:1397] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:35:08.777005 16443 raft_consensus.cc:348] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.777128 16443 raft_consensus.cc:374] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:08.777148 16443 raft_consensus.cc:729] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73242aa9c576446bb5fd27d69cca7af2, State: Initialized, Role: FOLLOWER
I20260927 18:35:08.777251 16443 consensus_queue.cc:260] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.777469 16443 ts_tablet_manager.cc:1428] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:35:08.777529 16155 heartbeater.cc:503] Master 127.5.151.254:34713 was elected leader, sending a full tablet report...
I20260927 18:35:08.797509 16445 raft_consensus.cc:482] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:35:08.797575 16445 raft_consensus.cc:504] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.797915 16445 leader_election.cc:288] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a2bc5d4cb9e4828b7ee20f64de31893 (127.5.151.194:45955), 4ef3c0b74746472e8058bfbdc8bb08a7 (127.5.151.195:38565)
I20260927 18:35:08.801574 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.798142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:38263 (local address 127.5.151.194:45955)
0927 18:35:08.798199 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.798200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.798209 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:35:08.798237 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.798238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.798244 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.798290 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:08.798448 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.799211 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.800658 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.801238 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.801315 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.801324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.801472 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.801515 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:35:08.801573 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.798039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:45955 (local address 127.5.151.193:38263)
0927 18:35:08.798101 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.798112 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.798152 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.798314 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.798315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.798317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:08.798381 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.798382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.799302 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.799302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.800553 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.800555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.801322 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.801323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.801399 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.801402 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.801448 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.801477 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":74}
I20260927 18:35:08.801654 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.798131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:52869 (local address 127.5.151.195:38565)
0927 18:35:08.798179 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.798181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.798337 (+ 156us) server_negotiation.cc:407] Connection header received
0927 18:35:08.798377 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.798378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.798384 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.798441 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:08.798599 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.799251 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.800661 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.801270 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.801326 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.801334 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.801545 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.801588 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:35:08.801889 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.798077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:38565 (local address 127.5.151.193:52869)
0927 18:35:08.798234 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.798249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:08.798324 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:08.798444 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:08.798444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:08.798446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:08.798502 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.798503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.799315 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.799315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.800553 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:08.800555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.801367 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.801368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:08.801468 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.801472 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:35:08.801526 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:35:08.801565 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260927 18:35:08.802048 16243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663a9c9b5ad14debaf8dd5d3e12c95c9" candidate_uuid: "73242aa9c576446bb5fd27d69cca7af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" is_pre_election: true
I20260927 18:35:08.802145 16377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663a9c9b5ad14debaf8dd5d3e12c95c9" candidate_uuid: "73242aa9c576446bb5fd27d69cca7af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" is_pre_election: true
I20260927 18:35:08.802184 16243 raft_consensus.cc:2454] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73242aa9c576446bb5fd27d69cca7af2 in term 0.
I20260927 18:35:08.802261 16377 raft_consensus.cc:2454] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73242aa9c576446bb5fd27d69cca7af2 in term 0.
I20260927 18:35:08.802374 16042 leader_election.cc:302] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73242aa9c576446bb5fd27d69cca7af2, 8a2bc5d4cb9e4828b7ee20f64de31893; no voters:
I20260927 18:35:08.802498 16445 raft_consensus.cc:2781] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:35:08.802541 16445 raft_consensus.cc:482] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:35:08.802615 16445 raft_consensus.cc:3037] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:08.803215 16445 raft_consensus.cc:504] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.803349 16445 leader_election.cc:288] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a2bc5d4cb9e4828b7ee20f64de31893 (127.5.151.194:45955), 4ef3c0b74746472e8058bfbdc8bb08a7 (127.5.151.195:38565)
I20260927 18:35:08.803498 16243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663a9c9b5ad14debaf8dd5d3e12c95c9" candidate_uuid: "73242aa9c576446bb5fd27d69cca7af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893"
I20260927 18:35:08.803516 16377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663a9c9b5ad14debaf8dd5d3e12c95c9" candidate_uuid: "73242aa9c576446bb5fd27d69cca7af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7"
I20260927 18:35:08.803572 16377 raft_consensus.cc:3037] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:08.803571 16243 raft_consensus.cc:3037] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:08.804375 16377 raft_consensus.cc:2454] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73242aa9c576446bb5fd27d69cca7af2 in term 1.
I20260927 18:35:08.804375 16243 raft_consensus.cc:2454] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73242aa9c576446bb5fd27d69cca7af2 in term 1.
I20260927 18:35:08.804544 16045 leader_election.cc:302] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ef3c0b74746472e8058bfbdc8bb08a7, 73242aa9c576446bb5fd27d69cca7af2; no voters:
I20260927 18:35:08.804641 16445 raft_consensus.cc:2781] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:08.804764 16445 raft_consensus.cc:686] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [term 1 LEADER]: Becoming Leader. State: Replica: 73242aa9c576446bb5fd27d69cca7af2, State: Running, Role: LEADER
I20260927 18:35:08.804847 16445 consensus_queue.cc:237] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } }
I20260927 18:35:08.805523 15963 catalog_manager.cc:5495] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73242aa9c576446bb5fd27d69cca7af2 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "73242aa9c576446bb5fd27d69cca7af2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73242aa9c576446bb5fd27d69cca7af2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 34329 } health_report { overall_health: HEALTHY } } }
I20260927 18:35:08.810389 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.808462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.5.151.193:34329)
0927 18:35:08.808495 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:08.808496 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:08.808504 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:35:08.808569 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:08.808570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:08.808582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:08.808640 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:35:08.808735 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.809133 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.809788 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:08.809968 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:08.810020 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:08.810148 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:35:08.810230 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:35:08.810285 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:08.810320 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 18:35:08.810335 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 18:35:08.811900 16243 raft_consensus.cc:1261] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 1 FOLLOWER]: Refusing update from remote peer 73242aa9c576446bb5fd27d69cca7af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:08.811900 16377 raft_consensus.cc:1261] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 1 FOLLOWER]: Refusing update from remote peer 73242aa9c576446bb5fd27d69cca7af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:08.812204 16447 consensus_queue.cc:1035] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ef3c0b74746472e8058bfbdc8bb08a7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:08.812333 16445 consensus_queue.cc:1035] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 73242aa9c576446bb5fd27d69cca7af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2bc5d4cb9e4828b7ee20f64de31893" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:08.814924 16452 mvcc.cc:205] Tried to move back new op lower bound from 7334027709691244544 to 7334027709665087488. Current Snapshot: MvccSnapshot[applied={T|T < 7334027709691244544}]
I20260927 18:35:08.815246 16451 mvcc.cc:205] Tried to move back new op lower bound from 7334027709691244544 to 7334027709665087488. Current Snapshot: MvccSnapshot[applied={T|T < 7334027709691244544 or (T in {7334027709691244544})}]
I20260927 18:35:08.815438 16450 mvcc.cc:205] Tried to move back new op lower bound from 7334027709691244544 to 7334027709665087488. Current Snapshot: MvccSnapshot[applied={T|T < 7334027709691244544 or (T in {7334027709691244544})}]
I20260927 18:35:08.816747 15962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:08.816828 15962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:08.817509 15962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 412b75cc259840d197f3d5a8ba743a43: Sending DeleteTablet for 3 replicas of tablet 663a9c9b5ad14debaf8dd5d3e12c95c9
I20260927 18:35:08.817793 16357 tablet_service.cc:1514] Processing DeleteTablet for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:08 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:41144
I20260927 18:35:08.817816 16089 tablet_service.cc:1514] Processing DeleteTablet for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:08 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:54728
I20260927 18:35:08.817827 16223 tablet_service.cc:1514] Processing DeleteTablet for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:08 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:58230
I20260927 18:35:08.818076 16461 tablet_replica.cc:330] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: stopping tablet replica
I20260927 18:35:08.818166 16461 raft_consensus.cc:2229] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:08.818235 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 16026
I20260927 18:35:08.818252 16461 raft_consensus.cc:2258] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:08.819095 16459 tablet_replica.cc:330] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: stopping tablet replica
I20260927 18:35:08.819190 16459 raft_consensus.cc:2229] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:08.819278 16459 raft_consensus.cc:2258] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:08.819557 16461 ts_tablet_manager.cc:1905] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:08.821327 16461 ts_tablet_manager.cc:1918] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:08.821385 16461 log.cc:1196] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/663a9c9b5ad14debaf8dd5d3e12c95c9
I20260927 18:35:08.821573 16461 ts_tablet_manager.cc:1939] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 8a2bc5d4cb9e4828b7ee20f64de31893: Deleting consensus metadata
I20260927 18:35:08.821982 15948 catalog_manager.cc:4841] TS 8a2bc5d4cb9e4828b7ee20f64de31893 (127.5.151.194:45955): tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 (table test-table [id=4889e6a783a840bea24bcbf1d5b6111b]) successfully deleted
I20260927 18:35:08.822840 16459 ts_tablet_manager.cc:1905] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:08.824002 16459 ts_tablet_manager.cc:1918] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:08.824091 16459 log.cc:1196] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/663a9c9b5ad14debaf8dd5d3e12c95c9
W20260927 18:35:08.824230 15948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:34329 (error 108)
I20260927 18:35:08.824388 16459 ts_tablet_manager.cc:1939] T 663a9c9b5ad14debaf8dd5d3e12c95c9 P 4ef3c0b74746472e8058bfbdc8bb08a7: Deleting consensus metadata
I20260927 18:35:08.824491 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 16160
I20260927 18:35:08.825785 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:08.825648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:34329 (local address 127.0.0.1:54754)
0927 18:35:08.825710 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:08.825750 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:34329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:35:08.825917 15948 catalog_manager.cc:4584] TS 73242aa9c576446bb5fd27d69cca7af2 (127.5.151.193:34329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 663a9c9b5ad14debaf8dd5d3e12c95c9: Network error: Client connection negotiation failed: client connection to 127.5.151.193:34329: connect: Connection refused (error 111)
I20260927 18:35:08.826709 15951 catalog_manager.cc:4841] TS 4ef3c0b74746472e8058bfbdc8bb08a7 (127.5.151.195:38565): tablet 663a9c9b5ad14debaf8dd5d3e12c95c9 (table test-table [id=4889e6a783a840bea24bcbf1d5b6111b]) successfully deleted
I20260927 18:35:08.829165 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 16294
I20260927 18:35:08.833709 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 15932
2026-09-27T18:35:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1765 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[16465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[16465](info): set up 2 sockets
Sep 27 18:35:08 dist-test-slave-19nl krb5kdc[16465](info): commencing operation
krb5kdc: starting...
W20260927 18:35:09.827589 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:35:09 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/rangeradmin_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/rangeradmin_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/rangerlookup_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/rangerlookup_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
I20260927 18:35:09.974779 5727 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskX68_uN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/postgres -l logfile start
2026-09-27 18:35:12.179 UTC [16495] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:35:12.179 UTC [16495] LOG: listening on IPv4 address "127.5.151.212", port 34449
2026-09-27 18:35:12.182 UTC [16495] LOG: listening on Unix socket "/tmp/.s.PGSQL.34449"
2026-09-27 18:35:12.219 UTC [16498] LOG: database system was shut down at 2026-09-27 18:35:10 UTC
2026-09-27 18:35:12.223 UTC [16495] LOG: database system is ready to accept connections
I20260927 18:35:13.155326 5727 mini_postgres.cc:96] Postgres bound to 34449
2026-09-27 18:35:13.161 UTC [16506] FATAL: database "slave" does not exist
127.5.151.212:34449 - accepting connections
I20260927 18:35:13.161880 5727 mini_ranger.cc:161] Starting Ranger...
I20260927 18:35:13.175565 5727 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:35:13.578015 5727 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:35:13.578104 5727 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-admin
2026-09-27 18:35:13,853 [I] DB FLAVOR :POSTGRES
2026-09-27 18:35:13,853 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:35:13,853 [I] Checking connection..
2026-09-27 18:35:13,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:35:14,104 [I] Checking connection passed.
2026-09-27 18:35:14,104 [I] --------- Verifying version history table ---------
2026-09-27 18:35:14,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:35:14,343 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:35:14,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:35:14,568 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:35:14,568 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:35:14,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:35:14.804 UTC [16570] WARNING: there is no transaction in progress
2026-09-27 18:35:14,810 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:35:14,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:35:15,067 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:35:15,068 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:35:15,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:35:15,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:35:15,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-19nl.c.gcp-upstream.internal','N') ;"
2026-09-27 18:35:15,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:35:16,041 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:35:16,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:35:16,283 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:35:16,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:35:16,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:35:16,769 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:35:16,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:35:17.658 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.672 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.686 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.700 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.714 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.729 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.780 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.786 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.800 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.810 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.819 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.832 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.840 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.846 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.852 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:17.858 UTC [16678] WARNING: there is no transaction in progress
2026-09-27 18:35:18,420 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:35:18,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:35:18,636 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:35:18,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:35:18,889 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:35:18,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:35:19,129 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:35:19,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:35:19,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:35:19,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-19nl.c.gcp-upstream.internal';"
2026-09-27 18:35:19,812 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:35:19,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:35:20,054 [I] DB_PATCHES have already been applied
I20260927 18:35:20.060045 5727 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-admin:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskX68_uN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:35:20.060099 5727 mini_ranger.cc:193] Using host: 127.5.151.212
I20260927 18:35:20.063093 5727 mini_ranger.cc:227] Ranger admin URL: http://127.5.151.212:33725
Sep 27, 2026 6:35:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:35:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:35:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:35:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:35:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:35:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:35:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:35:21.113462 5727 mini_ranger.cc:160] Time spent starting Ranger: real 7.952s user 0.000s sys 0.009s
Sep 27, 2026 6:35:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:35:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:35:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:35:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:35:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:35:40.845283 5727 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/rangerkms_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/rangerkms_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.151.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.151.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:35:40 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534140, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:35:40.926097 5727 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:35:40.934607 5727 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:35:41.657629 5727 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:35:41.657732 5727 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms
I20260927 18:35:41.657752 5727 mini_ranger_kms.cc:227] Using postgres at 127.5.151.212:34449
2026-09-27 18:35:41,822 [I] DB FLAVOR :POSTGRES
2026-09-27 18:35:41,822 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:35:41,822 [I] Checking connection
2026-09-27 18:35:41,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:35:42,070 [I] connection success
2026-09-27 18:35:42,070 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:35:42,071 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:35:42,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:35:42,304 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:35:42,304 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:35:42,304 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:35:42,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:34449/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:35:42,569 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:35:42.990386 5727 mini_ranger_kms.cc:310] Created kms service
I20260927 18:35:43.166050 5727 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:35:43.207947 5727 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:35:43 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534140, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:35:43.593192 5727 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:35:43.593309 5727 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms:/tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:35:43.593331 5727 mini_ranger_kms.cc:240] Using host: 127.5.151.212
I20260927 18:35:43.596879 5727 mini_ranger_kms.cc:276] Ranger KMS PID: 16858
I20260927 18:35:43.596985 5727 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.151.212:50429
Sep 27, 2026 6:35:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:35:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:35:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:35:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:35:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:35:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 18:35:45 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534145, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.151.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:45.761261 5727 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.835s user 0.003s sys 0.145s
I20260927 18:35:45.761370 5727 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:35:45.761456 5727 mini_ranger_kms.cc:401] 127.5.151.212:50429/kms/v1/keys
Sep 27 18:35:46 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534145, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.151.212@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
Sep 27 18:35:46 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534145, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.151.212@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
Sep 27 18:35:48 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:35:48.711437 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36731
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:36731
--ranger_config_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:50429/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:48.788887 16909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:48.789088 16909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:48.789119 16909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:48.790179 16909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:35:48.790238 16909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:48.790262 16909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:48.790289 16909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:35:48.790318 16909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:35:48.791800 16909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.151.212:50429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:36731
--ranger_config_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:48.792045 16909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:48.792273 16909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:48.797139 16914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:48.797176 16915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:48.797349 16917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:48.797200 16909 server_base.cc:1034] running on GCE node
I20260927 18:35:48.797770 16909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:48.798105 16909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:48.799259 16909 hybrid_clock.cc:648] HybridClock initialized: now 1790534148799236 us; error 41 us; skew 500 ppm
Sep 27 18:35:48 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534148, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:48.805648 16909 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:35:48.806046 16909 webserver.cc:460] Webserver started at http://127.5.151.254:38715/ using document root <none> and password file <none>
I20260927 18:35:48.806238 16909 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:48.806274 16909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:48.806406 16909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:48 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534148, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:35:48.849546 16909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "2946d2a20cca44ffbda8be3fa61373a2"
format_stamp: "Formatted at 2026-09-27 18:35:48 on dist-test-slave-19nl"
server_key: "3ed601473f309445bba8797afcd60b0d"
server_key_iv: "b9263bd80e331f065fbd6702579e4856"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:48.849989 16909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "2946d2a20cca44ffbda8be3fa61373a2"
format_stamp: "Formatted at 2026-09-27 18:35:48 on dist-test-slave-19nl"
server_key: "3ed601473f309445bba8797afcd60b0d"
server_key_iv: "b9263bd80e331f065fbd6702579e4856"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:48.851706 16909 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 18:35:48.865509 16924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:48.865664 16909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:35:48.865738 16909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "2946d2a20cca44ffbda8be3fa61373a2"
format_stamp: "Formatted at 2026-09-27 18:35:48 on dist-test-slave-19nl"
server_key: "3ed601473f309445bba8797afcd60b0d"
server_key_iv: "b9263bd80e331f065fbd6702579e4856"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:48.865821 16909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:48.885462 16909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:48.887465 16909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:48.887602 16909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:48.892416 16909 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:36731
I20260927 18:35:48.892438 16976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:36731 every 8 connection(s)
I20260927 18:35:48.892823 16909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:35:48.893473 16977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:48.895748 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2: Bootstrap starting.
I20260927 18:35:48.896311 16977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:48.896663 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 16909
I20260927 18:35:48.896657 16977 log.cc:824] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:48.896757 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:35:48.896971 5727 external_mini_cluster.cc:1428] Setting key 14fc2b6d151abe6f91825350d6fc2127
I20260927 18:35:48.897393 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2: No bootstrap required, opened a new log
I20260927 18:35:48.899804 16977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2946d2a20cca44ffbda8be3fa61373a2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36731 } }
I20260927 18:35:48.899971 16977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:48.900007 16977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2946d2a20cca44ffbda8be3fa61373a2, State: Initialized, Role: FOLLOWER
I20260927 18:35:48.900168 16977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2946d2a20cca44ffbda8be3fa61373a2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36731 } }
I20260927 18:35:48.900244 16977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:35:48.900270 16977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:35:48.900337 16977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:48.900970 16977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2946d2a20cca44ffbda8be3fa61373a2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36731 } }
I20260927 18:35:48.901082 16977 leader_election.cc:302] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2946d2a20cca44ffbda8be3fa61373a2; no voters:
I20260927 18:35:48.901252 16977 leader_election.cc:288] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:35:48.901276 16982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:48.901515 16982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [term 1 LEADER]: Becoming Leader. State: Replica: 2946d2a20cca44ffbda8be3fa61373a2, State: Running, Role: LEADER
I20260927 18:35:48.901600 16977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:35:48.901649 16982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2946d2a20cca44ffbda8be3fa61373a2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36731 } }
I20260927 18:35:48.903015 16982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2946d2a20cca44ffbda8be3fa61373a2. Latest consensus state: current_term: 1 leader_uuid: "2946d2a20cca44ffbda8be3fa61373a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2946d2a20cca44ffbda8be3fa61373a2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36731 } } }
I20260927 18:35:48.903167 16982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:35:48.903240 16977 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:35:48.903596 16996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:35:48.904058 16996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:35:48.904624 16983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2946d2a20cca44ffbda8be3fa61373a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2946d2a20cca44ffbda8be3fa61373a2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36731 } } }
I20260927 18:35:48.904779 16983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2 [sys.catalog]: This master's current role is: LEADER
Sep 27 18:35:48 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:48.910804 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:48.897454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.5.151.254:36731)
0927 18:35:48.897611 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:48.897619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:48.897648 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:35:48.898509 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:48.898518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:48.898590 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:48.898726 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:48.899474 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:48.900823 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:48.901934 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:48.902643 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:48.902779 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:48.906696 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:48.906741 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:48.906741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:48.906813 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:48.909033 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:48.909412 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:48.909415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:48.909415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:48.909518 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:48.909712 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:48.909713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:48.909713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:48.910010 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:48.910140 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:48.910261 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:35:48.910592 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260927 18:35:48.921525 16996 catalog_manager.cc:1347] Generated new cluster ID: daf37904cab04d759069b58441238de4
I20260927 18:35:48.921593 16996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:35:48.942219 16996 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:35:48.943140 16996 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:35:48.952329 16996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2: Generated new TSK 0
I20260927 18:35:48.952643 16996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:35:49.597844 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:50429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36731
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:49.698338 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:49.698539 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:49.698637 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:49.699862 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:49.699904 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:49.699931 17022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:35:49.701395 17022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.151.212:50429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:49.701632 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:49.701882 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:49.707204 17027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:49.707206 17028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:49.708163 17022 server_base.cc:1034] running on GCE node
W20260927 18:35:49.710758 17030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:49.710975 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:49.711236 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:49.712430 17022 hybrid_clock.cc:648] HybridClock initialized: now 1790534149712401 us; error 40 us; skew 500 ppm
Sep 27 18:35:49 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:49.715613 17022 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:35:49.716167 17022 webserver.cc:460] Webserver started at http://127.5.151.193:38025/ using document root <none> and password file <none>
I20260927 18:35:49.716440 17022 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:49.716509 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:49.716648 17022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:49 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:35:49.730878 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "a50eb29e0a394cf28c9315fb82b315e4"
format_stamp: "Formatted at 2026-09-27 18:35:49 on dist-test-slave-19nl"
server_key: "911d585096b268219bb3b8ee40e96d85"
server_key_iv: "72abf84879562b9a4eabbc48307713cf"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:49.731326 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "a50eb29e0a394cf28c9315fb82b315e4"
format_stamp: "Formatted at 2026-09-27 18:35:49 on dist-test-slave-19nl"
server_key: "911d585096b268219bb3b8ee40e96d85"
server_key_iv: "72abf84879562b9a4eabbc48307713cf"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:49.733147 17022 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:35:49.751075 17039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:49.751302 17022 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260927 18:35:49.751408 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "a50eb29e0a394cf28c9315fb82b315e4"
format_stamp: "Formatted at 2026-09-27 18:35:49 on dist-test-slave-19nl"
server_key: "911d585096b268219bb3b8ee40e96d85"
server_key_iv: "72abf84879562b9a4eabbc48307713cf"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:49.751531 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:49.766535 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:49.769193 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:49.769356 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:49.769670 17022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:49.770056 17022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:49.770108 17022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:49.770139 17022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:49.770188 17022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:49.777020 17022 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:40995
I20260927 18:35:49.777284 17152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:40995 every 8 connection(s)
I20260927 18:35:49.777532 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:35:49 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:49.785301 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 17022
I20260927 18:35:49.785404 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:35:49.785666 5727 external_mini_cluster.cc:1428] Setting key bb37727abc98420bb19992c46ac347af
I20260927 18:35:49.789230 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:49.779695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:58441 (local address 127.5.151.254:36731)
0927 18:35:49.780497 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:49.780500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:49.780508 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:35:49.780538 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:49.780539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:49.780553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:49.780630 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:49.781285 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:49.782116 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:49.783168 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:49.783469 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:49.783551 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:49.785716 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:49.785717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:49.785717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:49.785726 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:49.787621 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:49.788207 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:49.788208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:49.788209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:49.788284 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:49.788544 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:49.788545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:49.788546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:49.788671 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:49.788741 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:49.789038 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 18:35:49.789129 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":67,"threads_started":1}
I20260927 18:35:49.789448 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:49.779370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36731 (local address 127.5.151.193:58441)
0927 18:35:49.779608 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:49.779659 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:49.780292 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:49.780759 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:49.780765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:49.780986 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:49.781183 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:49.781189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:49.782211 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:49.782212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:49.783082 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:49.783084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:49.783555 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:49.783555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:49.783652 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:49.784192 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:49.784201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:49.785597 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:49.787730 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:49.787736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:49.787739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:49.788105 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:49.788339 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:49.788340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:49.788340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:49.788471 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:49.788745 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:49.788752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:49.788902 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 18:35:49.789017 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:35:49.789178 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":128,"threads_started":1}
I20260927 18:35:49.790031 17153 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36731
I20260927 18:35:49.790148 17153 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:49.790355 17153 heartbeater.cc:511] Master 127.5.151.254:36731 requested a full tablet report, sending...
I20260927 18:35:49.790925 16941 ts_manager.cc:194] Registered new tserver with Master: a50eb29e0a394cf28c9315fb82b315e4 (127.5.151.193:40995)
I20260927 18:35:49.791612 16941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:58441
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:35:49.825953 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:50429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36731
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:35:49 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:35:49.938942 17159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:49.939208 17159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:49.939244 17159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:49.940869 17159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:49.940976 17159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:49.941046 17159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:35:49.943218 17159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.151.212:50429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:49.943447 17159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:49.943784 17159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:49.950088 17169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:49.950285 17166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:49.950488 17167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:49.951488 17159 server_base.cc:1034] running on GCE node
I20260927 18:35:49.951684 17159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:49.951942 17159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:49.953097 17159 hybrid_clock.cc:648] HybridClock initialized: now 1790534149953065 us; error 43 us; skew 500 ppm
Sep 27 18:35:49 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:49.956377 17159 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:35:49.956876 17159 webserver.cc:460] Webserver started at http://127.5.151.194:38575/ using document root <none> and password file <none>
I20260927 18:35:49.957167 17159 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:49.957247 17159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:49.957424 17159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:49 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:35:49.970400 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "6741721b6ab6450fb8e7b40b408f1ba5"
format_stamp: "Formatted at 2026-09-27 18:35:49 on dist-test-slave-19nl"
server_key: "0a11a530b2a50a1c431678d614f3198f"
server_key_iv: "a836c5f78fc53df005b0f4f217b3019d"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:49.970945 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "6741721b6ab6450fb8e7b40b408f1ba5"
format_stamp: "Formatted at 2026-09-27 18:35:49 on dist-test-slave-19nl"
server_key: "0a11a530b2a50a1c431678d614f3198f"
server_key_iv: "a836c5f78fc53df005b0f4f217b3019d"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:49.972833 17159 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:35:49.994223 17177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:49.994565 17159 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 18:35:49.994768 17159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "6741721b6ab6450fb8e7b40b408f1ba5"
format_stamp: "Formatted at 2026-09-27 18:35:49 on dist-test-slave-19nl"
server_key: "0a11a530b2a50a1c431678d614f3198f"
server_key_iv: "a836c5f78fc53df005b0f4f217b3019d"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:49.994961 17159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:50.012168 17159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:50.014266 17159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:50.014413 17159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:50.014715 17159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:50.015080 17159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:50.015127 17159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:50.015189 17159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:50.015218 17159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:50.024497 17159 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:37145
I20260927 18:35:50.024991 17159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:35:50.025516 17290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:37145 every 8 connection(s)
I20260927 18:35:50.026300 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 17159
I20260927 18:35:50.026389 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:35:50.026604 5727 external_mini_cluster.cc:1428] Setting key 203b8f1a988f2036693c52fc3ed933a5
Sep 27 18:35:50 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:50.037434 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.026658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:32885 (local address 127.5.151.254:36731)
0927 18:35:50.026784 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.026787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.027030 (+ 243us) server_negotiation.cc:407] Connection header received
0927 18:35:50.027900 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.027901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.027916 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.027960 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:50.028787 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.029552 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.031084 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.031384 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.031463 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.034031 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:50.034032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:50.034033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:50.034040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:50.035822 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.036318 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.036319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.036319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.036376 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.036677 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.036678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.036678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.036806 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:50.036852 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.037250 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.037338 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 18:35:50.038198 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.026152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36731 (local address 127.5.151.194:32885)
0927 18:35:50.026905 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.026959 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:50.027780 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:50.028135 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:50.028140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:50.028389 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:50.028669 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.028675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.029636 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.029637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.030987 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.030990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.031532 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.031534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.031680 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.032249 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:50.032261 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:50.033898 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:50.035920 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:50.035938 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:50.035940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:50.036218 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:50.036456 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:50.036456 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:50.036456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:50.036592 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:50.036903 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:50.036910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:50.037090 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 18:35:50.037785 (+ 695us) client_negotiation.cc:240] Negotiation successful
0927 18:35:50.037962 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
I20260927 18:35:50.038941 17291 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36731
I20260927 18:35:50.039043 17291 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:50.039237 17291 heartbeater.cc:511] Master 127.5.151.254:36731 requested a full tablet report, sending...
I20260927 18:35:50.039598 16941 ts_manager.cc:194] Registered new tserver with Master: 6741721b6ab6450fb8e7b40b408f1ba5 (127.5.151.194:37145)
I20260927 18:35:50.040086 16941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:32885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:35:50.054841 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:50429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36731
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:35:50.165954 17296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:35:50.166183 17296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:35:50.166220 17296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:35:50.167428 17296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:35:50.167490 17296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:35:50.167538 17296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:35:50.169145 17296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.151.212:50429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:36731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:35:50.169591 17296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:35:50.169898 17296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:35:50.176087 17302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:50.176412 17301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:35:50.176669 17304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:35:50.177647 17296 server_base.cc:1034] running on GCE node
I20260927 18:35:50.177852 17296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:35:50.178125 17296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:35:50.179280 17296 hybrid_clock.cc:648] HybridClock initialized: now 1790534150179260 us; error 35 us; skew 500 ppm
Sep 27 18:35:50 dist-test-slave-19nl krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534150, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:35:50.182052 17296 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:35:50.182531 17296 webserver.cc:460] Webserver started at http://127.5.151.195:43057/ using document root <none> and password file <none>
I20260927 18:35:50.182843 17296 fs_manager.cc:359] Metadata directory not provided
I20260927 18:35:50.182942 17296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:35:50.183104 17296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:35:50 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534150, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:35:50.194785 17296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "409d944122a44c1fa78c259ff3880e5a"
format_stamp: "Formatted at 2026-09-27 18:35:50 on dist-test-slave-19nl"
server_key: "23f504c2a9c225dfea5af80a04e45d47"
server_key_iv: "57789217e7681338706496a8402664b8"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:50.195271 17296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "409d944122a44c1fa78c259ff3880e5a"
format_stamp: "Formatted at 2026-09-27 18:35:50 on dist-test-slave-19nl"
server_key: "23f504c2a9c225dfea5af80a04e45d47"
server_key_iv: "57789217e7681338706496a8402664b8"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:50.203449 17296 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260927 18:35:50.224192 17311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:50.224476 17296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:35:50.224640 17296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "409d944122a44c1fa78c259ff3880e5a"
format_stamp: "Formatted at 2026-09-27 18:35:50 on dist-test-slave-19nl"
server_key: "23f504c2a9c225dfea5af80a04e45d47"
server_key_iv: "57789217e7681338706496a8402664b8"
server_key_version: "kuduclusterkey@0"
I20260927 18:35:50.224815 17296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:35:50.250902 17296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:35:50.253314 17296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:35:50.253501 17296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:35:50.253760 17296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:35:50.254174 17296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:35:50.254262 17296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:50.254331 17296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:35:50.254401 17296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:35:50.260658 17296 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:43775
I20260927 18:35:50.261133 17296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:35:50.262053 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 17296
I20260927 18:35:50.262161 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:35:50.262348 5727 external_mini_cluster.cc:1428] Setting key 09df2ee883e80ff5c070d2202ece776d
I20260927 18:35:50.274946 17424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:43775 every 8 connection(s)
Sep 27 18:35:50 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534150, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:35:50.286455 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.275455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:51853 (local address 127.5.151.254:36731)
0927 18:35:50.275495 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.275498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.275695 (+ 197us) server_negotiation.cc:407] Connection header received
0927 18:35:50.276389 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.276390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.276403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.276439 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:50.277284 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.278080 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.279655 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.279953 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.279996 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.282766 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:50.282767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:50.282767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:50.282775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:50.284394 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.285355 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.285356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.285356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.285417 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.285738 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.285739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.285739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.285867 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:50.285910 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.286287 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.286363 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:35:50.287166 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.275161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:36731 (local address 127.5.151.195:51853)
0927 18:35:50.275567 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.275613 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:50.276250 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:50.276634 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:50.276639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:50.276888 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:35:50.277176 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.277182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.278198 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.278199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.279557 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.279560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.280098 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.280099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.280252 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.280858 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:35:50.280867 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:35:50.282654 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:35:50.284674 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:50.284680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:50.284682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:50.285267 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:50.285494 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:35:50.285495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:35:50.285495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:35:50.285638 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:35:50.285957 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:35:50.285963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:35:50.286127 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 18:35:50.286770 (+ 643us) client_negotiation.cc:240] Negotiation successful
0927 18:35:50.286925 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":102,"threads_started":1}
I20260927 18:35:50.287806 17425 heartbeater.cc:347] Connected to a master server at 127.5.151.254:36731
I20260927 18:35:50.287894 17425 heartbeater.cc:464] Registering TS with master...
I20260927 18:35:50.288054 17425 heartbeater.cc:511] Master 127.5.151.254:36731 requested a full tablet report, sending...
I20260927 18:35:50.288424 16941 ts_manager.cc:194] Registered new tserver with Master: 409d944122a44c1fa78c259ff3880e5a (127.5.151.195:43775)
I20260927 18:35:50.288908 16941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:51853
I20260927 18:35:50.295872 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:35:50.315682 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.297242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.5.151.254:36731)
0927 18:35:50.297277 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.297279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.308145 (+ 10866us) server_negotiation.cc:407] Connection header received
0927 18:35:50.308245 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.308246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.308257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.308286 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:35:50.309037 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.309741 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.310838 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.311117 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.311160 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.312695 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:35:50.312696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:35:50.312697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:35:50.312703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:35:50.314421 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.314848 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.314849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.314850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.314904 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:35:50.315116 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:35:50.315117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:35:50.315117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:35:50.315231 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:35:50.315313 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.315521 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.315596 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:35:50.317794 16941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:35:50.320155 16941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:35:50.337350 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.327580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:37145 (local address 127.0.0.1:51732)
0927 18:35:50.328678 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.328717 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:50.329048 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:50.329397 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:50.329400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:50.329402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:50.329483 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.329485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.330942 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.330943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.332193 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.332196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.337080 (+ 4884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.337081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.337155 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.337168 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 18:35:50.337234 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:35:50.337280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":63,"threads_started":1}
I20260927 18:35:50.341585 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.328329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.5.151.194:37145)
0927 18:35:50.328932 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.328936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.328947 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:35:50.329176 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.329179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.329196 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.329322 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:50.329561 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.330852 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.333955 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.334752 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.334821 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.334902 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.341460 (+ 6558us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.341504 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":79,"threads_started":1}
I20260927 18:35:50.342343 17225 tablet_service.cc:1467] Processing CreateTablet for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 (DEFAULT_TABLE table=test-table [id=01ea29c60bf2493882e03807e89dabf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:50.342717 17225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:50.346419 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.327783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:43775 (local address 127.0.0.1:49910)
0927 18:35:50.332304 (+ 4521us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.332312 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:50.332350 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:50.332608 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:50.332609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:50.332611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:50.332647 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.332648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.334937 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.334938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.336053 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.336056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.346193 (+ 10137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.346194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.346260 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.346264 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:35:50.346317 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:35:50.346353 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4506,"thread_start_us":62,"threads_started":1}
I20260927 18:35:50.346781 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.328101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.5.151.195:43775)
0927 18:35:50.328859 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.328864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.332403 (+ 3539us) server_negotiation.cc:407] Connection header received
0927 18:35:50.332463 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.332465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.332480 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.332543 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:50.332711 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.333818 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.336168 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.336906 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.336975 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.337044 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.346653 (+ 9609us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.346695 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":69,"threads_started":1}
I20260927 18:35:50.347406 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.327913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:40995 (local address 127.0.0.1:44430)
0927 18:35:50.337753 (+ 9840us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.337761 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:50.337795 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:50.338054 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:50.338055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:50.338056 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:50.338085 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.338086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.339373 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.339374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.340472 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.340475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.347218 (+ 6743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.347219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.347272 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.347275 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:35:50.347320 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 18:35:50.347352 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9830,"thread_start_us":51,"threads_started":1}
I20260927 18:35:50.347550 17359 tablet_service.cc:1467] Processing CreateTablet for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 (DEFAULT_TABLE table=test-table [id=01ea29c60bf2493882e03807e89dabf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:50.347703 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.328507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.5.151.193:40995)
0927 18:35:50.329006 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.329009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.337851 (+ 8842us) server_negotiation.cc:407] Connection header received
0927 18:35:50.337899 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.337901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.337917 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.337982 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:50.338153 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.339246 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.340569 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.341302 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.341370 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.341418 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.347591 (+ 6173us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.347634 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":43,"threads_started":1}
I20260927 18:35:50.347841 17359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:50.348315 17087 tablet_service.cc:1467] Processing CreateTablet for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 (DEFAULT_TABLE table=test-table [id=01ea29c60bf2493882e03807e89dabf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:35:50.348614 17087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:35:50.351229 17444 tablet_bootstrap.cc:492] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Bootstrap starting.
I20260927 18:35:50.352002 17443 tablet_bootstrap.cc:492] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Bootstrap starting.
I20260927 18:35:50.352221 17444 tablet_bootstrap.cc:654] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:50.352653 17444 log.cc:824] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:50.352977 17445 tablet_bootstrap.cc:492] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4: Bootstrap starting.
I20260927 18:35:50.353025 17443 tablet_bootstrap.cc:654] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:50.353645 17445 tablet_bootstrap.cc:654] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4: Neither blocks nor log segments found. Creating new log.
I20260927 18:35:50.353909 17445 log.cc:824] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:50.354705 17444 tablet_bootstrap.cc:492] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: No bootstrap required, opened a new log
I20260927 18:35:50.354761 17444 ts_tablet_manager.cc:1397] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 18:35:50.356706 17444 raft_consensus.cc:348] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.356954 17444 raft_consensus.cc:374] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:50.357129 17444 raft_consensus.cc:729] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409d944122a44c1fa78c259ff3880e5a, State: Initialized, Role: FOLLOWER
I20260927 18:35:50.357321 17444 consensus_queue.cc:260] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.357692 17444 ts_tablet_manager.cc:1428] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:50.357836 17425 heartbeater.cc:503] Master 127.5.151.254:36731 was elected leader, sending a full tablet report...
I20260927 18:35:50.353329 17443 log.cc:824] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:35:50.360041 17445 tablet_bootstrap.cc:492] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4: No bootstrap required, opened a new log
I20260927 18:35:50.360100 17445 ts_tablet_manager.cc:1397] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 18:35:50.362466 17445 raft_consensus.cc:348] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.362629 17445 raft_consensus.cc:374] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:50.362663 17445 raft_consensus.cc:729] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50eb29e0a394cf28c9315fb82b315e4, State: Initialized, Role: FOLLOWER
I20260927 18:35:50.362761 17445 consensus_queue.cc:260] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.362977 17445 ts_tablet_manager.cc:1428] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:35:50.363368 17153 heartbeater.cc:503] Master 127.5.151.254:36731 was elected leader, sending a full tablet report...
I20260927 18:35:50.364028 17443 tablet_bootstrap.cc:492] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: No bootstrap required, opened a new log
I20260927 18:35:50.364141 17443 ts_tablet_manager.cc:1397] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260927 18:35:50.366191 17443 raft_consensus.cc:348] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.366402 17443 raft_consensus.cc:374] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:35:50.366480 17443 raft_consensus.cc:729] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6741721b6ab6450fb8e7b40b408f1ba5, State: Initialized, Role: FOLLOWER
I20260927 18:35:50.366658 17443 consensus_queue.cc:260] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.366946 17443 ts_tablet_manager.cc:1428] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 18:35:50.367090 17291 heartbeater.cc:503] Master 127.5.151.254:36731 was elected leader, sending a full tablet report...
Sep 27 18:35:50 dist-test-slave-19nl krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534149, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:35:50.413321 17448 raft_consensus.cc:482] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:35:50.413465 17448 raft_consensus.cc:504] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.413911 17448 leader_election.cc:288] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6741721b6ab6450fb8e7b40b408f1ba5 (127.5.151.194:37145), a50eb29e0a394cf28c9315fb82b315e4 (127.5.151.193:40995)
I20260927 18:35:50.418203 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.414172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:37145 (local address 127.5.151.195:38151)
0927 18:35:50.414624 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.414637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:50.414679 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:50.415125 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:50.415127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:50.415129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:50.415191 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.415193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.416044 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.416045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.417183 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.417185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.417959 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.417960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.418046 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.418050 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:35:50.418097 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 18:35:50.418136 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430}
I20260927 18:35:50.418427 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.414806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:38151 (local address 127.5.151.194:37145)
0927 18:35:50.414899 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.414901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.414908 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:35:50.414934 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.414936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.414946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.415025 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:50.415265 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.415927 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.417311 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.417860 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.417922 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.417931 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.418303 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.418353 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:35:50.418789 17245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b90cb78400f49b0af86cfe5f0f4e3f9" candidate_uuid: "409d944122a44c1fa78c259ff3880e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" is_pre_election: true
I20260927 18:35:50.418952 17245 raft_consensus.cc:2454] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409d944122a44c1fa78c259ff3880e5a in term 0.
I20260927 18:35:50.419165 17313 leader_election.cc:302] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409d944122a44c1fa78c259ff3880e5a, 6741721b6ab6450fb8e7b40b408f1ba5; no voters:
I20260927 18:35:50.419281 17448 raft_consensus.cc:2781] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:35:50.419317 17448 raft_consensus.cc:482] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:35:50.419337 17448 raft_consensus.cc:3037] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:50.420284 17448 raft_consensus.cc:504] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.420403 17448 leader_election.cc:288] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [CANDIDATE]: Term 1 election: Requested vote from peers 6741721b6ab6450fb8e7b40b408f1ba5 (127.5.151.194:37145), a50eb29e0a394cf28c9315fb82b315e4 (127.5.151.193:40995)
I20260927 18:35:50.420743 17245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b90cb78400f49b0af86cfe5f0f4e3f9" candidate_uuid: "409d944122a44c1fa78c259ff3880e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6741721b6ab6450fb8e7b40b408f1ba5"
I20260927 18:35:50.420828 17245 raft_consensus.cc:3037] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:50.421731 17245 raft_consensus.cc:2454] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409d944122a44c1fa78c259ff3880e5a in term 1.
I20260927 18:35:50.421918 17313 leader_election.cc:302] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409d944122a44c1fa78c259ff3880e5a, 6741721b6ab6450fb8e7b40b408f1ba5; no voters:
I20260927 18:35:50.422045 17448 raft_consensus.cc:2781] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:35:50.422186 17448 raft_consensus.cc:686] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 1 LEADER]: Becoming Leader. State: Replica: 409d944122a44c1fa78c259ff3880e5a, State: Running, Role: LEADER
I20260927 18:35:50.422273 17448 consensus_queue.cc:237] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } }
I20260927 18:35:50.423095 16941 catalog_manager.cc:5495] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 409d944122a44c1fa78c259ff3880e5a (127.5.151.195). New cstate: current_term: 1 leader_uuid: "409d944122a44c1fa78c259ff3880e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "409d944122a44c1fa78c259ff3880e5a" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 } health_report { overall_health: UNKNOWN } } }
I20260927 18:35:50.426364 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.414859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:42373 (local address 127.5.151.193:40995)
0927 18:35:50.415092 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.415094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.416923 (+ 1829us) server_negotiation.cc:407] Connection header received
0927 18:35:50.417053 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.417055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.417063 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.417090 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:35:50.417311 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.417984 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.424939 (+ 6955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.425449 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.425503 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.425512 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.426266 (+ 754us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.426305 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260927 18:35:50.426373 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.414270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:40995 (local address 127.5.151.195:42373)
0927 18:35:50.416857 (+ 2587us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.416869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:35:50.416948 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:35:50.417145 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:35:50.417146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:35:50.417148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:35:50.417232 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.417234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.418063 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.418064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.424840 (+ 6776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:35:50.424843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.426115 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.426116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:35:50.426183 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.426186 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:35:50.426260 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:35:50.426297 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2569,"mutex_wait_us":12,"thread_start_us":101,"threads_started":1}
I20260927 18:35:50.426966 17107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b90cb78400f49b0af86cfe5f0f4e3f9" candidate_uuid: "409d944122a44c1fa78c259ff3880e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50eb29e0a394cf28c9315fb82b315e4" is_pre_election: true
I20260927 18:35:50.427114 17107 raft_consensus.cc:2454] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409d944122a44c1fa78c259ff3880e5a in term 0.
I20260927 18:35:50.427222 17106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b90cb78400f49b0af86cfe5f0f4e3f9" candidate_uuid: "409d944122a44c1fa78c259ff3880e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50eb29e0a394cf28c9315fb82b315e4"
I20260927 18:35:50.427273 17106 raft_consensus.cc:3037] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:35:50.428118 17106 raft_consensus.cc:2454] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409d944122a44c1fa78c259ff3880e5a in term 1.
I20260927 18:35:50.447171 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.443876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.5.151.195:43775)
0927 18:35:50.443928 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:35:50.443931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:35:50.443939 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:35:50.444033 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:35:50.444034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:35:50.444052 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:35:50.444106 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:35:50.444333 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.445038 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.446125 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:35:50.446403 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:35:50.446469 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:35:50.446718 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:35:50.446830 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:35:50.446892 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:35:50.447085 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 18:35:50.447103 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:35:50.449129 17106 raft_consensus.cc:1261] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P a50eb29e0a394cf28c9315fb82b315e4 [term 1 FOLLOWER]: Refusing update from remote peer 409d944122a44c1fa78c259ff3880e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:50.449453 17453 consensus_queue.cc:1035] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a50eb29e0a394cf28c9315fb82b315e4" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:50.451403 17245 raft_consensus.cc:1261] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 1 FOLLOWER]: Refusing update from remote peer 409d944122a44c1fa78c259ff3880e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:35:50.451646 17453 consensus_queue.cc:1035] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6741721b6ab6450fb8e7b40b408f1ba5" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:35:50.455610 17456 mvcc.cc:205] Tried to move back new op lower bound from 7334027880236761088 to 7334027880130187264. Current Snapshot: MvccSnapshot[applied={T|T < 7334027880236761088}]
I20260927 18:35:50.455793 17459 mvcc.cc:205] Tried to move back new op lower bound from 7334027880236761088 to 7334027880130187264. Current Snapshot: MvccSnapshot[applied={T|T < 7334027880236761088}]
I20260927 18:35:50.457937 17457 mvcc.cc:205] Tried to move back new op lower bound from 7334027880236761088 to 7334027880130187264. Current Snapshot: MvccSnapshot[applied={T|T < 7334027880236761088}]
I20260927 18:35:50.460666 16941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:50.460759 16941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:35:50.461416 16941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2946d2a20cca44ffbda8be3fa61373a2: Sending DeleteTablet for 3 replicas of tablet 3b90cb78400f49b0af86cfe5f0f4e3f9
I20260927 18:35:50.461928 17225 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:50 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:51732
I20260927 18:35:50.462154 17359 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:50 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:49910
I20260927 18:35:50.462360 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:35:50 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:44430
I20260927 18:35:50.462771 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 17022
I20260927 18:35:50.466634 17465 tablet_replica.cc:330] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: stopping tablet replica
I20260927 18:35:50.466742 17465 raft_consensus.cc:2229] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:35:50.466845 17465 raft_consensus.cc:2258] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:50.467079 17466 tablet_replica.cc:330] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: stopping tablet replica
I20260927 18:35:50.467173 17466 raft_consensus.cc:2229] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:35:50.467283 17466 raft_consensus.cc:2258] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:35:50.467593 17465 ts_tablet_manager.cc:1905] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:50.473150 17466 ts_tablet_manager.cc:1905] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:35:50.474815 17466 ts_tablet_manager.cc:1918] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:50.474890 17466 log.cc:1196] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/3b90cb78400f49b0af86cfe5f0f4e3f9
I20260927 18:35:50.475153 17466 ts_tablet_manager.cc:1939] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 409d944122a44c1fa78c259ff3880e5a: Deleting consensus metadata
I20260927 18:35:50.482637 16926 catalog_manager.cc:4841] TS 409d944122a44c1fa78c259ff3880e5a (127.5.151.195:43775): tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 (table test-table [id=01ea29c60bf2493882e03807e89dabf0]) successfully deleted
I20260927 18:35:50.482667 17465 ts_tablet_manager.cc:1918] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:35:50.482744 17465 log.cc:1196] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/3b90cb78400f49b0af86cfe5f0f4e3f9
I20260927 18:35:50.483425 17465 ts_tablet_manager.cc:1939] T 3b90cb78400f49b0af86cfe5f0f4e3f9 P 6741721b6ab6450fb8e7b40b408f1ba5: Deleting consensus metadata
W20260927 18:35:50.483932 16926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:40995 (error 108)
I20260927 18:35:50.484202 16926 catalog_manager.cc:4841] TS 6741721b6ab6450fb8e7b40b408f1ba5 (127.5.151.194:37145): tablet 3b90cb78400f49b0af86cfe5f0f4e3f9 (table test-table [id=01ea29c60bf2493882e03807e89dabf0]) successfully deleted
I20260927 18:35:50.486676 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 17159
I20260927 18:35:50.493891 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 17296
I20260927 18:35:50.496675 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:35:50.496539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:40995 (local address 127.0.0.1:44432)
0927 18:35:50.496595 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 18:35:50.496639 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:35:50.496797 16926 catalog_manager.cc:4584] TS a50eb29e0a394cf28c9315fb82b315e4 (127.5.151.193:40995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b90cb78400f49b0af86cfe5f0f4e3f9: Network error: Client connection negotiation failed: client connection to 127.5.151.193:40995: connect: Connection refused (error 111)
I20260927 18:35:50.501144 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 16909
I20260927 18:35:50.508198 5727 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:35:50.852185 5727 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:35:50.852269 5727 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:35:51.195806 5727 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:35:51.195 UTC [16495] LOG: received smart shutdown request
2026-09-27 18:35:51.198 UTC [16495] LOG: background worker "logical replication launcher" (PID 16504) exited with exit code 1
2026-09-27 18:35:51.199 UTC [16499] LOG: shutting down
2026-09-27 18:35:51.360 UTC [16495] LOG: database system is shut down
2026-09-27T18:35:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42629 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:35:51 dist-test-slave-19nl krb5kdc[17477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:35:51 dist-test-slave-19nl krb5kdc[17477](info): set up 2 sockets
Sep 27 18:35:51 dist-test-slave-19nl krb5kdc[17477](info): commencing operation
krb5kdc: starting...
W20260927 18:35:52.458519 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:35:52 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:35:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:35:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/rangeradmin_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/rangeradmin_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/rangerlookup_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/rangerlookup_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
I20260927 18:35:52.602905 5727 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskX68_uN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/postgres -l logfile start
2026-09-27 18:35:54.795 UTC [17507] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:35:54.795 UTC [17507] LOG: listening on IPv4 address "127.5.151.212", port 48555
2026-09-27 18:35:54.798 UTC [17507] LOG: listening on Unix socket "/tmp/.s.PGSQL.48555"
2026-09-27 18:35:54.819 UTC [17510] LOG: database system was shut down at 2026-09-27 18:35:53 UTC
2026-09-27 18:35:54.822 UTC [17507] LOG: database system is ready to accept connections
I20260927 18:35:55.730162 5727 mini_postgres.cc:96] Postgres bound to 48555
2026-09-27 18:35:55.735 UTC [17518] FATAL: database "slave" does not exist
127.5.151.212:48555 - accepting connections
I20260927 18:35:55.736440 5727 mini_ranger.cc:161] Starting Ranger...
I20260927 18:35:55.750314 5727 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:35:56.149855 5727 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:35:56.149947 5727 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-admin
2026-09-27 18:35:56,413 [I] DB FLAVOR :POSTGRES
2026-09-27 18:35:56,414 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:35:56,414 [I] Checking connection..
2026-09-27 18:35:56,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:35:56,643 [I] Checking connection passed.
2026-09-27 18:35:56,643 [I] --------- Verifying version history table ---------
2026-09-27 18:35:56,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:35:56,866 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:35:56,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:35:57,087 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:35:57,087 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:35:57,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:35:57.323 UTC [17582] WARNING: there is no transaction in progress
2026-09-27 18:35:57,329 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:35:57,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:35:57,576 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:35:57,576 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:35:57,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:35:57,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:35:58,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-19nl.c.gcp-upstream.internal','N') ;"
2026-09-27 18:35:58,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:35:58,480 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:35:58,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:35:58,716 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:35:58,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:35:58,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:35:59,147 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:35:59,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:36:00.040 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.055 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.070 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.083 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.100 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.119 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.180 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.185 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.199 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.208 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.217 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.230 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.238 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.244 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.250 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00.256 UTC [17690] WARNING: there is no transaction in progress
2026-09-27 18:36:00,803 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:36:00,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:36:01,038 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:36:01,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:36:01,263 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:36:01,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:36:01,485 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:36:01,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:36:01,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:36:01,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-19nl.c.gcp-upstream.internal';"
2026-09-27 18:36:02,197 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:36:02,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:36:02,413 [I] DB_PATCHES have already been applied
I20260927 18:36:02.418351 5727 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-admin:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskX68_uN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:36:02.418429 5727 mini_ranger.cc:193] Using host: 127.5.151.212
I20260927 18:36:02.421401 5727 mini_ranger.cc:227] Ranger admin URL: http://127.5.151.212:34969
Sep 27, 2026 6:36:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:36:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:36:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:36:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:36:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:36:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:36:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:36:03.468487 5727 mini_ranger.cc:160] Time spent starting Ranger: real 7.732s user 0.001s sys 0.007s
Sep 27, 2026 6:36:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:36:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:36:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:36:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:36:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:36:24.179926 5727 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.151.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.151.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/rangerkms_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.151.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/rangerkms_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.151.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.151.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.151.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/HTTP_127.5.151.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:36:24 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534184, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:36:24.261893 5727 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:36:24.270124 5727 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:36:25.077533 5727 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:36:25.077594 5727 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms
I20260927 18:36:25.077606 5727 mini_ranger_kms.cc:227] Using postgres at 127.5.151.212:48555
2026-09-27 18:36:25,237 [I] DB FLAVOR :POSTGRES
2026-09-27 18:36:25,237 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:36:25,237 [I] Checking connection
2026-09-27 18:36:25,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:36:25,472 [I] connection success
2026-09-27 18:36:25,472 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:36:25,472 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:36:25,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:36:25,691 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:36:25,691 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:36:25,691 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:36:25,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.151.212:48555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:36:25,926 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:36:26.366291 5727 mini_ranger_kms.cc:310] Created kms service
I20260927 18:36:26.521998 5727 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:36:26.564710 5727 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:36:26 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534184, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:26.954662 5727 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:36:26.954869 5727 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms:/tmp/dist-test-taskX68_uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskX68_uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:36:26.954892 5727 mini_ranger_kms.cc:240] Using host: 127.5.151.212
I20260927 18:36:26.958161 5727 mini_ranger_kms.cc:276] Ranger KMS PID: 17870
I20260927 18:36:26.958238 5727 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.151.212:54435
Sep 27, 2026 6:36:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:36:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:36:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:36:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:36:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:36:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 18:36:29 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534189, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.151.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:29.134541 5727 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.873s user 0.000s sys 0.144s
I20260927 18:36:29.134699 5727 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:36:29.134753 5727 mini_ranger_kms.cc:401] 127.5.151.212:54435/kms/v1/keys
Sep 27 18:36:29 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534189, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.151.212@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
Sep 27 18:36:29 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534189, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.151.212@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
Sep 27 18:36:31 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:36:31.917435 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41639
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41639
--ranger_config_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:32.016958 17921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:32.017162 17921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:32.017185 17921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:32.018290 17921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:32.018329 17921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:32.018343 17921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:32.018355 17921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:32.018368 17921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:32.019928 17921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.151.212:54435/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41639
--ranger_config_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:32.020152 17921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:32.020414 17921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:32.026170 17926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:32.026199 17927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:32.026247 17921 server_base.cc:1034] running on GCE node
W20260927 18:36:32.026356 17929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:32.026682 17921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:32.026980 17921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:32.028160 17921 hybrid_clock.cc:648] HybridClock initialized: now 1790534192028141 us; error 33 us; skew 500 ppm
Sep 27 18:36:32 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:32.030930 17921 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:32.031359 17921 webserver.cc:460] Webserver started at http://127.5.151.254:35049/ using document root <none> and password file <none>
I20260927 18:36:32.031572 17921 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:32.031633 17921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:32.031767 17921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:36:32 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:32.088729 17921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "2f919aee429e4e6984219cd1a3968ff7"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "da36be4de861d4a2c81e75d9abe715c3"
server_key_iv: "77d5ae2f5303ce8ed017d6f51087e9b9"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:32.089123 17921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "2f919aee429e4e6984219cd1a3968ff7"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "da36be4de861d4a2c81e75d9abe715c3"
server_key_iv: "77d5ae2f5303ce8ed017d6f51087e9b9"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:32.090504 17921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:36:32.103921 17936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:32.104091 17921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:36:32.104189 17921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "2f919aee429e4e6984219cd1a3968ff7"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "da36be4de861d4a2c81e75d9abe715c3"
server_key_iv: "77d5ae2f5303ce8ed017d6f51087e9b9"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:32.104285 17921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:32.126928 17921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:32.128926 17921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:32.129091 17921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:32.133826 17921 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:41639
I20260927 18:36:32.133914 17988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:41639 every 8 connection(s)
I20260927 18:36:32.134230 17921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:32.134946 17989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:32.137102 17989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Bootstrap starting.
I20260927 18:36:32.137676 17989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:32.137961 17989 log.cc:824] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:32.138751 17989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: No bootstrap required, opened a new log
I20260927 18:36:32.140641 17989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:32.140787 17989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:32.140810 17989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f919aee429e4e6984219cd1a3968ff7, State: Initialized, Role: FOLLOWER
I20260927 18:36:32.140960 17989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:32.141026 17989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:32.141078 17989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:32.141121 17989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:32.141633 17989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:32.141722 17989 leader_election.cc:302] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f919aee429e4e6984219cd1a3968ff7; no voters:
I20260927 18:36:32.141887 17989 leader_election.cc:288] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:36:32.141908 17992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:32.142021 17992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 2f919aee429e4e6984219cd1a3968ff7, State: Running, Role: LEADER
I20260927 18:36:32.142131 17989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:32.142158 17992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:32.142496 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f919aee429e4e6984219cd1a3968ff7. Latest consensus state: current_term: 1 leader_uuid: "2f919aee429e4e6984219cd1a3968ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } } }
I20260927 18:36:32.142514 17994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f919aee429e4e6984219cd1a3968ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } } }
I20260927 18:36:32.142638 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:32.142730 17994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:32.143483 18000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:32.143631 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 17921
I20260927 18:36:32.143734 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:36:32.143920 5727 external_mini_cluster.cc:1428] Setting key f01c9467c24bfe88e2345ff381cd3fe9
I20260927 18:36:32.144025 18000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:32.144881 17989 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:36:32.152361 18000 catalog_manager.cc:1347] Generated new cluster ID: 40287da369814b92807f1cafe263edb4
I20260927 18:36:32.152426 18000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:36:32 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:32.158638 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:32.144571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.5.151.254:41639)
0927 18:36:32.147287 (+ 2716us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:32.147301 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:32.147354 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:36:32.147624 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:32.147635 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:32.147757 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:32.147969 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:32.149233 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:32.150284 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:32.151376 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:32.152065 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:32.152171 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:32.154983 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:32.154987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:32.154987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:32.154998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:32.156828 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:32.157165 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:32.157168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:32.157168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:32.157252 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:32.157409 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:32.157410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:32.157410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:32.157729 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:32.157857 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:32.158295 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 18:36:32.158407 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2532,"thread_start_us":62,"threads_started":1}
I20260927 18:36:32.170269 18000 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:36:32.171090 18000 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:32.179687 18000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Generated new TSK 0
I20260927 18:36:32.179921 18000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:36:32.845121 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41639
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:32.953650 18034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:32.953820 18034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:32.953840 18034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:32.955240 18034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:32.955295 18034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:32.955333 18034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:36:32.957463 18034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.151.212:54435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:32.957695 18034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:32.957968 18034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:32.963096 18040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:32.963136 18039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:32.963251 18042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:32.963265 18034 server_base.cc:1034] running on GCE node
I20260927 18:36:32.963567 18034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:32.963855 18034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:32.965008 18034 hybrid_clock.cc:648] HybridClock initialized: now 1790534192964996 us; error 26 us; skew 500 ppm
Sep 27 18:36:32 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:32.967824 18034 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:36:32.968302 18034 webserver.cc:460] Webserver started at http://127.5.151.193:38853/ using document root <none> and password file <none>
I20260927 18:36:32.968534 18034 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:32.968609 18034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:32.968739 18034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:36:32 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:32.982317 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "141c1ed3bbb2430c958a840135d8d9c3"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "b59bfc129b8ffc079211f2317b5b1795"
server_key_iv: "bfa998ef1110ae19513973d84dc74ede"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:32.982755 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "141c1ed3bbb2430c958a840135d8d9c3"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "b59bfc129b8ffc079211f2317b5b1795"
server_key_iv: "bfa998ef1110ae19513973d84dc74ede"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:32.984583 18034 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:36:32.998374 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:32.998543 18034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:32.998714 18034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "141c1ed3bbb2430c958a840135d8d9c3"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "b59bfc129b8ffc079211f2317b5b1795"
server_key_iv: "bfa998ef1110ae19513973d84dc74ede"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:32.998879 18034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:33.022637 18034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:33.025063 18034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:33.025254 18034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:33.025522 18034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:33.026908 18034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:33.026968 18034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.027050 18034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:33.027091 18034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.034088 18034 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:40785
I20260927 18:36:33.034257 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:40785 every 8 connection(s)
I20260927 18:36:33.034608 18034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534192, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:33.042655 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 18034
I20260927 18:36:33.042747 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:36:33.043273 5727 external_mini_cluster.cc:1428] Setting key 9fb1d638b1a5d62db83bd81b51713dbf
I20260927 18:36:33.046186 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.035350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41639 (local address 127.5.151.193:42591)
0927 18:36:33.035599 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:33.035644 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:33.036296 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:33.036621 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:33.036626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:33.036844 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:33.037121 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.037127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.038119 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.038121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.039388 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.039391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.039912 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.039913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.040089 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.040627 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:33.040638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:33.042329 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:33.044494 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.044500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.044502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.044777 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.044995 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.044996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.044996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.045141 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.045516 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:33.045522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:33.045677 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 18:36:33.045802 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 18:36:33.045949 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260927 18:36:33.046618 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.035389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:42591 (local address 127.5.151.254:41639)
0927 18:36:33.035536 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:33.035538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:33.035751 (+ 213us) server_negotiation.cc:407] Connection header received
0927 18:36:33.036426 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:33.036428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:33.036443 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:33.036548 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:33.037233 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.038016 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.039484 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.039799 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.039872 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.042429 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:33.042430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:33.042431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:33.042457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:33.044367 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.044862 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.044864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.044864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.044925 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.045222 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.045223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.045224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.045353 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:33.045486 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:33.046430 (+ 944us) server_negotiation.cc:299] Negotiation successful
0927 18:36:33.046508 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260927 18:36:33.047066 18165 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41639
I20260927 18:36:33.047155 18165 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:33.047327 18165 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:33.047920 17953 ts_manager.cc:194] Registered new tserver with Master: 141c1ed3bbb2430c958a840135d8d9c3 (127.5.151.193:40785)
I20260927 18:36:33.048825 17953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:42591
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:36:33.067497 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41639
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:36:33.188285 18171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:33.188589 18171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:33.188665 18171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:33.190302 18171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:33.190392 18171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:33.190460 18171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:36:33.192526 18171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.151.212:54435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:33.192915 18171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:33.193212 18171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:33.199393 18181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.199592 18179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.199734 18178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:33.200421 18171 server_base.cc:1034] running on GCE node
I20260927 18:36:33.200676 18171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:33.200911 18171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:33.202068 18171 hybrid_clock.cc:648] HybridClock initialized: now 1790534193202048 us; error 31 us; skew 500 ppm
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:33.205130 18171 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:36:33.205593 18171 webserver.cc:460] Webserver started at http://127.5.151.194:42381/ using document root <none> and password file <none>
I20260927 18:36:33.205816 18171 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:33.205864 18171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:33.205974 18171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:33.218505 18171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "e2f21dca85d54f8497d48d91c1716645"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "1299da3335fef283cf08da018812c969"
server_key_iv: "5a78557f7aaa678ecc08403d96916d7f"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.219050 18171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "e2f21dca85d54f8497d48d91c1716645"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "1299da3335fef283cf08da018812c969"
server_key_iv: "5a78557f7aaa678ecc08403d96916d7f"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.220762 18171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:36:33.231830 18188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.232051 18171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:36:33.232152 18171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "e2f21dca85d54f8497d48d91c1716645"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "1299da3335fef283cf08da018812c969"
server_key_iv: "5a78557f7aaa678ecc08403d96916d7f"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.232245 18171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:33.242928 18171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:33.245036 18171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:33.245193 18171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:33.245446 18171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:33.245810 18171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:33.245913 18171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.245954 18171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:33.246006 18171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.253063 18171 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:41639
I20260927 18:36:33.253229 18302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:41639 every 8 connection(s)
I20260927 18:36:33.253485 18171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:36:33.257655 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 18171
I20260927 18:36:33.257766 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:36:33.257952 5727 external_mini_cluster.cc:1428] Setting key 38b3f0191fd4d8a9e522f02ba238e343
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:33.265914 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.254194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41639 (local address 127.5.151.194:50117)
0927 18:36:33.254422 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:33.254452 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:33.254986 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:33.255250 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:33.255253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:33.255426 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:33.255619 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.255624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.256309 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.256311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.257094 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.257096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.258160 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.258162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.258658 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.260680 (+ 2022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:33.260697 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:33.262128 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:33.264189 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.264195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.264197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.264487 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.264716 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.264716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.264717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.264857 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.265165 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:33.265171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:33.265344 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 18:36:33.265459 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:36:33.265640 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260927 18:36:33.266677 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.254240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:50117 (local address 127.5.151.254:41639)
0927 18:36:33.254298 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:33.254300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:33.254574 (+ 274us) server_negotiation.cc:407] Connection header received
0927 18:36:33.255108 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:33.255109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:33.255124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:33.255175 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:33.255711 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.256216 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.257191 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.257402 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.257486 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.262222 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:33.262224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:33.262224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:33.262234 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:33.264094 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.264583 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.264584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.264585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.264641 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.264931 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.264932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.264932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.265069 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:33.265136 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:33.266435 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0927 18:36:33.266562 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:36:33.267211 18303 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41639
I20260927 18:36:33.267350 18303 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:33.267583 18303 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:33.267982 17953 ts_manager.cc:194] Registered new tserver with Master: e2f21dca85d54f8497d48d91c1716645 (127.5.151.194:41639)
I20260927 18:36:33.268358 17953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:50117
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:36:33.284508 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41639
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:33.399441 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:33.399777 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:33.399865 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:33.401536 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:33.401640 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:33.401719 18308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:36:33.403857 18308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.151.212:54435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:33.404271 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:33.404587 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:33.410874 18314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.411209 18313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.411458 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:33.412595 18308 server_base.cc:1034] running on GCE node
I20260927 18:36:33.412793 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:33.413049 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:33.414243 18308 hybrid_clock.cc:648] HybridClock initialized: now 1790534193414224 us; error 36 us; skew 500 ppm
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:33.417639 18308 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:36:33.418157 18308 webserver.cc:460] Webserver started at http://127.5.151.195:35681/ using document root <none> and password file <none>
I20260927 18:36:33.418448 18308 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:33.418588 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:33.418776 18308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:33.435613 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "1e47cf1202a04ada8cbfa66e81557c15"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "7bf6b2270c17e3733e9ca513e2f2ea45"
server_key_iv: "73e77310f62688020f885dfc25403988"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.439981 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "1e47cf1202a04ada8cbfa66e81557c15"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "7bf6b2270c17e3733e9ca513e2f2ea45"
server_key_iv: "73e77310f62688020f885dfc25403988"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.441882 18308 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:36:33.450934 18323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.451149 18308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 18:36:33.451233 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "1e47cf1202a04ada8cbfa66e81557c15"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "7bf6b2270c17e3733e9ca513e2f2ea45"
server_key_iv: "73e77310f62688020f885dfc25403988"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.451297 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:33.469897 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:33.472172 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:33.472337 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:33.472573 18308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:33.472922 18308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:33.473001 18308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.473065 18308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:33.473117 18308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.479946 18308 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:43801
I20260927 18:36:33.480327 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:33.482103 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 18308
I20260927 18:36:33.482206 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:36:33.482372 5727 external_mini_cluster.cc:1428] Setting key 51dc980d263dc95914b68f39c8d8c06f
I20260927 18:36:33.481902 18436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:43801 every 8 connection(s)
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:33.490986 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.481018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:45955 (local address 127.5.151.254:41639)
0927 18:36:33.481063 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:33.481065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:33.481493 (+ 428us) server_negotiation.cc:407] Connection header received
0927 18:36:33.482149 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:33.482151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:33.482164 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:33.482193 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:33.482980 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.483676 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.485071 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.485376 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.485452 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.487821 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:33.487822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:33.487823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:33.487829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:33.489461 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.489930 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.489931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.489931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.489990 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.490260 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.490261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.490261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.490385 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:33.490426 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:33.490824 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 18:36:33.490904 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:36:33.491688 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.481030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41639 (local address 127.5.151.195:45955)
0927 18:36:33.481365 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:33.481407 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:33.482018 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:33.482394 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:33.482397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:33.482642 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:33.482879 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.482885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.483803 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.483803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.484971 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.484974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.485485 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.485486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.485616 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.486194 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:33.486204 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:33.487718 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:33.489553 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.489559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.489561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.489836 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.490064 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.490064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.490064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.490183 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.490470 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:33.490475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:33.490667 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 18:36:33.491309 (+ 642us) client_negotiation.cc:240] Negotiation successful
0927 18:36:33.491460 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":71,"threads_started":1}
I20260927 18:36:33.492350 18437 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41639
I20260927 18:36:33.492432 18437 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:33.492594 18437 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:33.492923 17953 ts_manager.cc:194] Registered new tserver with Master: 1e47cf1202a04ada8cbfa66e81557c15 (127.5.151.195:43801)
I20260927 18:36:33.493363 17953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:45955
I20260927 18:36:33.496909 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:36:33.497776 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 18034
I20260927 18:36:33.507615 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 18171
I20260927 18:36:33.514408 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 18308
I20260927 18:36:33.519328 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 17921
I20260927 18:36:33.523350 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41639
--webserver_interface=127.5.151.254
--webserver_port=35049
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41639
--ranger_config_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:33.600605 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:33.600853 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:33.600888 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:33.601986 18440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:33.602047 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:33.602074 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:33.602104 18440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:33.602131 18440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:33.603679 18440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41639
--ranger_config_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=35049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:33.603919 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:33.604131 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:33.608611 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.608603 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.608667 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:33.608820 18440 server_base.cc:1034] running on GCE node
I20260927 18:36:33.609053 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:33.609326 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:33.610486 18440 hybrid_clock.cc:648] HybridClock initialized: now 1790534193610476 us; error 35 us; skew 500 ppm
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:33.612706 18440 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:33.613108 18440 webserver.cc:460] Webserver started at http://127.5.151.254:35049/ using document root <none> and password file <none>
I20260927 18:36:33.613325 18440 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:33.613394 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:33.680331 18440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:33.681023 18456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.681193 18440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:33.681300 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "2f919aee429e4e6984219cd1a3968ff7"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "da36be4de861d4a2c81e75d9abe715c3"
server_key_iv: "77d5ae2f5303ce8ed017d6f51087e9b9"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.681701 18440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:33.703082 18440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:33.704545 18440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:33.704702 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:33.708726 18440 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:41639
I20260927 18:36:33.708772 18508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:41639 every 8 connection(s)
I20260927 18:36:33.709137 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:33.710242 18509 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:36:33.710983 18509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Bootstrap starting.
I20260927 18:36:33.718408 18509 log.cc:824] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:33.718777 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 18440
I20260927 18:36:33.719071 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:40785
--local_ip_for_outbound_sockets=127.5.151.193
--tserver_master_addrs=127.5.151.254:41639
--webserver_port=38853
--webserver_interface=127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:33.720148 18509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:36:33.720413 18509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Bootstrap complete.
I20260927 18:36:33.722755 18509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:33.723003 18509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f919aee429e4e6984219cd1a3968ff7, State: Initialized, Role: FOLLOWER
I20260927 18:36:33.723205 18509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:33.723302 18509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:33.723353 18509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:33.723402 18509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:36:33.724365 18509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:33.724483 18509 leader_election.cc:302] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f919aee429e4e6984219cd1a3968ff7; no voters:
I20260927 18:36:33.724659 18509 leader_election.cc:288] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:36:33.724709 18513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:36:33.724841 18513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [term 2 LEADER]: Becoming Leader. State: Replica: 2f919aee429e4e6984219cd1a3968ff7, State: Running, Role: LEADER
I20260927 18:36:33.724916 18509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:33.724989 18513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } }
I20260927 18:36:33.725333 18514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f919aee429e4e6984219cd1a3968ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } } }
I20260927 18:36:33.725347 18515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f919aee429e4e6984219cd1a3968ff7. Latest consensus state: current_term: 2 leader_uuid: "2f919aee429e4e6984219cd1a3968ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f919aee429e4e6984219cd1a3968ff7" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41639 } } }
I20260927 18:36:33.725430 18514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:33.725449 18515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:33.725641 18520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:33.726177 18520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:33.726369 18520 catalog_manager.cc:1259] Loaded cluster ID: 40287da369814b92807f1cafe263edb4
I20260927 18:36:33.726436 18520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:36:33.726581 18509 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:36:33.727587 18520 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:33.727773 18520 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Loaded TSK: 0
I20260927 18:36:33.727988 18520 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:36:33.817824 18512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:33.818029 18512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:33.818061 18512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:33.819262 18512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:33.819324 18512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:33.819367 18512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:36:33.821054 18512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:40785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=38853
--tserver_master_addrs=127.5.151.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:33.821291 18512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:33.821522 18512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:33.826761 18546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.826771 18547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.827056 18549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:33.827310 18512 server_base.cc:1034] running on GCE node
I20260927 18:36:33.827491 18512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:33.827734 18512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:33.828876 18512 hybrid_clock.cc:648] HybridClock initialized: now 1790534193828865 us; error 25 us; skew 500 ppm
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:33.831208 18512 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:36:33.831663 18512 webserver.cc:460] Webserver started at http://127.5.151.193:38853/ using document root <none> and password file <none>
I20260927 18:36:33.831890 18512 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:33.831966 18512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:33.851877 18512 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:33.852766 18557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.852949 18512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 18:36:33.853164 18512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "141c1ed3bbb2430c958a840135d8d9c3"
format_stamp: "Formatted at 2026-09-27 18:36:32 on dist-test-slave-19nl"
server_key: "b59bfc129b8ffc079211f2317b5b1795"
server_key_iv: "bfa998ef1110ae19513973d84dc74ede"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:33.853607 18512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:33.873039 18512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:33.875293 18512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:33.875432 18512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:33.875636 18512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:33.876112 18512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:33.876179 18512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.876221 18512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:33.876286 18512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:33.883339 18512 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:40785
I20260927 18:36:33.883419 18670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:40785 every 8 connection(s)
I20260927 18:36:33.883733 18512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:36:33.887217 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 18512
I20260927 18:36:33.887506 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:41639
--local_ip_for_outbound_sockets=127.5.151.194
--tserver_master_addrs=127.5.151.254:41639
--webserver_port=42381
--webserver_interface=127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:36:33 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534193, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:33.895502 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.884360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41639 (local address 127.5.151.193:37471)
0927 18:36:33.884560 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:33.884602 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:33.885186 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:33.885619 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:33.885623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:33.885808 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:33.886067 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.886072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.887384 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.887385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.888604 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:33.888607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.889493 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.889494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:33.889620 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.890226 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:33.890236 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:33.891781 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:33.893644 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.893650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.893652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.893941 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.894254 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:33.894254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:33.894255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:33.894377 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:33.894893 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:33.894899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:33.895032 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 18:36:33.895151 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 18:36:33.895298 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":120,"threads_started":1}
I20260927 18:36:33.896345 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:33.884494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:37471 (local address 127.5.151.254:41639)
0927 18:36:33.884701 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:33.884709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:33.884746 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:36:33.885395 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:33.885400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:33.885436 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:33.885532 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:33.886201 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.887255 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.888704 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:33.889292 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:33.889388 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:33.891874 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:33.891878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:33.891878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:33.891888 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:33.893553 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.894035 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.894038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.894038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.894176 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:33.894452 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:33.894453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:33.894453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:33.894791 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:33.894866 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:33.895991 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0927 18:36:33.896125 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260927 18:36:33.897377 18671 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41639
I20260927 18:36:33.897461 18671 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:33.897626 18671 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
W20260927 18:36:33.898579 18671 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:41639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:36:33.898655 18671 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:33.898729 18671 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:33.899125 18671 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:33.899209 18671 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
W20260927 18:36:33.899528 18671 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:36:33.987921 18675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:33.988250 18675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:33.988324 18675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:33.990110 18675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:33.990231 18675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:33.990315 18675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:36:33.992429 18675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=42381
--tserver_master_addrs=127.5.151.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:33.992882 18675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:33.993198 18675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:33.998596 18681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.998596 18683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:33.998615 18680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:33.999254 18675 server_base.cc:1034] running on GCE node
I20260927 18:36:33.999485 18675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:33.999768 18675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:34.000924 18675 hybrid_clock.cc:648] HybridClock initialized: now 1790534194000914 us; error 29 us; skew 500 ppm
Sep 27 18:36:34 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:34.003588 18675 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:36:34.004032 18675 webserver.cc:460] Webserver started at http://127.5.151.194:42381/ using document root <none> and password file <none>
I20260927 18:36:34.004298 18675 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:34.004370 18675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:36:34 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:34.026186 18675 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260927 18:36:34.031163 18691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:34.031335 18675 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260927 18:36:34.031423 18675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "e2f21dca85d54f8497d48d91c1716645"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "1299da3335fef283cf08da018812c969"
server_key_iv: "5a78557f7aaa678ecc08403d96916d7f"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:34.031801 18675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:34.046897 18675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:34.049070 18675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:34.049239 18675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:34.049470 18675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:34.049865 18675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:34.049901 18675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:34.049921 18675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:34.049932 18675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:34.056679 18675 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:41639
I20260927 18:36:34.056771 18804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:41639 every 8 connection(s)
I20260927 18:36:34.057163 18675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:36:34.057610 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 18675
I20260927 18:36:34.057884 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.151.212:54435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:43801
--local_ip_for_outbound_sockets=127.5.151.195
--tserver_master_addrs=127.5.151.254:41639
--webserver_port=35681
--webserver_interface=127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:36:34 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:34.079357 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:34.063192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:39081 (local address 127.5.151.254:41639)
0927 18:36:34.063265 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:34.063267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:34.064621 (+ 1354us) server_negotiation.cc:407] Connection header received
0927 18:36:34.070702 (+ 6081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:34.070703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:34.070720 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:34.070766 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:34.071479 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.072122 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.073494 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.073811 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.073854 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:34.076223 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:34.076225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:34.076225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:34.076231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:34.077892 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:34.078366 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:34.078367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:34.078367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:34.078428 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:34.078726 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:34.078727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:34.078727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:34.078852 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:34.078927 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:34.079189 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 18:36:34.079269 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:36:34.079517 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:34.062878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41639 (local address 127.5.151.194:39081)
0927 18:36:34.064296 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:34.064525 (+ 229us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:34.070583 (+ 6058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:34.070884 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:34.070889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:34.071118 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:34.071372 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:34.071379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.072251 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.072252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:34.073397 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:34.073400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.073964 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.073965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:34.074108 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:34.074748 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:34.074757 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:34.076137 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:34.077984 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:34.077990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:34.077992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:34.078268 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:34.078491 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:34.078493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:34.078493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:34.078659 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:34.078919 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:34.078926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:34.079057 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 18:36:34.079156 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:36:34.079290 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1380,"thread_start_us":100,"threads_started":1}
I20260927 18:36:34.080027 18805 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41639
I20260927 18:36:34.080137 18805 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:34.080332 18805 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
W20260927 18:36:34.080960 18805 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:41639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:36:34.081041 18805 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:34.081123 18805 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:34.081455 18805 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:34.081532 18805 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
W20260927 18:36:34.081794 18805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:36:34.176968 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:34.177284 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:34.177337 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:34.178938 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:34.179041 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:34.179141 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:36:34.181087 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.151.212:54435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:43801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=35681
--tserver_master_addrs=127.5.151.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:34.181281 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:34.181561 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:34.186821 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:34.186933 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:34.186811 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:34.187037 18807 server_base.cc:1034] running on GCE node
I20260927 18:36:34.187256 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:34.187441 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:34.188640 18807 hybrid_clock.cc:648] HybridClock initialized: now 1790534194188623 us; error 29 us; skew 500 ppm
Sep 27 18:36:34 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:34.191705 18807 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:36:34.192202 18807 webserver.cc:460] Webserver started at http://127.5.151.195:35681/ using document root <none> and password file <none>
I20260927 18:36:34.192451 18807 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:34.192550 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:36:34 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:34.207173 18807 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:36:34.211007 18824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:34.211208 18807 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 18:36:34.211300 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "1e47cf1202a04ada8cbfa66e81557c15"
format_stamp: "Formatted at 2026-09-27 18:36:33 on dist-test-slave-19nl"
server_key: "7bf6b2270c17e3733e9ca513e2f2ea45"
server_key_iv: "73e77310f62688020f885dfc25403988"
server_key_version: "kuduclusterkey@0"
I20260927 18:36:34.211740 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:34.231873 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:34.233948 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:34.234122 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:34.234376 18807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:34.234792 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:34.234841 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:34.234870 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:34.234896 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:34.242403 18807 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:43801
I20260927 18:36:34.242897 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:34.243216 18938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:43801 every 8 connection(s)
I20260927 18:36:34.247643 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 18807
Sep 27 18:36:34 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:34.253515 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:34.243631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:44963 (local address 127.5.151.254:41639)
0927 18:36:34.243694 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:34.243697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:34.243942 (+ 245us) server_negotiation.cc:407] Connection header received
0927 18:36:34.244613 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:34.244614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:34.244627 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:34.244687 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:34.245353 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.246105 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.247566 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.247844 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.247893 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:34.250323 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:34.250325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:34.250325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:34.250332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:34.251938 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:34.252403 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:34.252404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:34.252405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:34.252468 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:34.252784 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:34.252785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:34.252786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:34.252911 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:34.252989 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:34.253320 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 18:36:34.253403 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:36:34.253692 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:34.243640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41639 (local address 127.5.151.195:44963)
0927 18:36:34.243823 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:34.243869 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:34.244470 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:34.244775 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:34.244780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:34.244980 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:34.245239 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:34.245245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.246207 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.246208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:34.247459 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:34.247462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.247999 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.248000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:34.248138 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:34.248640 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:34.248650 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:34.250229 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:34.252030 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:34.252036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:34.252038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:34.252323 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:34.252568 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:34.252569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:34.252570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:34.252698 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:34.253000 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:34.253006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:34.253160 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 18:36:34.253283 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 18:36:34.253449 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260927 18:36:34.254259 18939 heartbeater.cc:347] Connected to a master server at 127.5.151.254:41639
I20260927 18:36:34.254355 18939 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:34.254592 18939 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
W20260927 18:36:34.255741 18939 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:41639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:36:34.255807 18939 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:34.255862 18939 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:34.256203 18939 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:34.256275 18939 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
W20260927 18:36:34.256659 18939 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:36:34.257138 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:34.248007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.5.151.254:41639)
0927 18:36:34.248106 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:34.248108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:34.248223 (+ 115us) server_negotiation.cc:407] Connection header received
0927 18:36:34.248333 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:34.248334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:34.248345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:34.248398 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:34.249165 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.249886 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.251094 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:34.251382 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:34.251455 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:34.253536 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:34.253537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:34.253537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:34.253544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:34.255440 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:34.255988 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:34.255989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:34.255989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:34.256047 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:34.256427 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:34.256428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:34.256429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:34.256588 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:34.256874 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:34.256950 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:36:34.257031 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
W20260927 18:36:34.710531 18504 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.5.151.254:41639
Sep 27 18:36:34 dist-test-slave-19nl krb5kdc[17477](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:34.899706 18671 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:34.899849 18671 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:34.900254 18473 ts_manager.cc:194] Registered new tserver with Master: 141c1ed3bbb2430c958a840135d8d9c3 (127.5.151.193:40785)
I20260927 18:36:34.901189 18473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:37471
W20260927 18:36:35.058025 18800 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.5.151.194:41639
I20260927 18:36:35.081959 18805 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:35.082126 18805 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:35.082597 18473 ts_manager.cc:194] Registered new tserver with Master: e2f21dca85d54f8497d48d91c1716645 (127.5.151.194:41639)
I20260927 18:36:35.083040 18473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:39081
Sep 27 18:36:35 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
I20260927 18:36:35.256821 18939 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:35.256973 18939 heartbeater.cc:511] Master 127.5.151.254:41639 requested a full tablet report, sending...
I20260927 18:36:35.257283 18473 ts_manager.cc:194] Registered new tserver with Master: 1e47cf1202a04ada8cbfa66e81557c15 (127.5.151.195:43801)
I20260927 18:36:35.257774 18473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:44963
I20260927 18:36:35.263372 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:36:35.278646 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.264277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.5.151.254:41639)
0927 18:36:35.266144 (+ 1867us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:35.266147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:35.266291 (+ 144us) server_negotiation.cc:407] Connection header received
0927 18:36:35.266342 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:35.266343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:35.266356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:35.266427 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:35.267110 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.267849 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.268833 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.269135 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.269205 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.275593 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:35.275594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:35.275595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:35.275602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:35.277371 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:35.277794 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:35.277795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:35.277796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:35.277857 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:35.278096 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:35.278096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:35.278097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:35.278219 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:35.278313 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:35.278454 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:36:35.278535 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1843,"thread_start_us":55,"threads_started":1}
I20260927 18:36:35.280800 18473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:36:35.282891 18473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:36:35.290712 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.286215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:43801 (local address 127.0.0.1:47512)
0927 18:36:35.286336 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:35.286346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:35.286393 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:35.286828 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:35.286831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:35.286834 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:35.286897 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.286899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.288185 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.288186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.289445 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.289447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.290392 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.290393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.290483 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.290502 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:36:35.290585 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:36:35.290637 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260927 18:36:35.291100 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.286472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.5.151.195:43801)
0927 18:36:35.286650 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:35.286655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:35.286664 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:36:35.286697 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:35.286700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:35.286714 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:35.286976 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:35.287054 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.288079 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.289562 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.290290 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.290898 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.290964 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:35.290991 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 18:36:35.291027 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":46,"threads_started":1}
I20260927 18:36:35.291754 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.286340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:41639 (local address 127.0.0.1:33842)
0927 18:36:35.286480 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:35.286488 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:35.286522 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:35.287271 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:35.287272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:35.287273 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:35.287324 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.287325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.288585 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.288586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.289900 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.289903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.291549 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.291550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.291621 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.291626 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:36:35.291654 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 18:36:35.291687 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260927 18:36:35.291806 18873 tablet_service.cc:1467] Processing CreateTablet for tablet 15311bdaaf564bf88ac8703f16b57861 (DEFAULT_TABLE table=test-table [id=be39023a9b5d46e2ab0a59295a483d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:35.292152 18873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15311bdaaf564bf88ac8703f16b57861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:35.292248 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.286719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.5.151.194:41639)
0927 18:36:35.287114 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:35.287120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:35.287129 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:36:35.287161 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:35.287163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:35.287177 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:35.287386 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:35.287457 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.288489 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.290001 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.290748 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.292042 (+ 1294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.292110 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:35.292134 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 18:36:35.292171 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":49,"threads_started":1}
I20260927 18:36:35.292791 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 15311bdaaf564bf88ac8703f16b57861 (DEFAULT_TABLE table=test-table [id=be39023a9b5d46e2ab0a59295a483d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:35.293108 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15311bdaaf564bf88ac8703f16b57861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:35.296137 18973 tablet_bootstrap.cc:492] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Bootstrap starting.
I20260927 18:36:35.296820 18973 tablet_bootstrap.cc:654] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:35.297132 18973 log.cc:824] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:35.297907 18973 tablet_bootstrap.cc:492] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: No bootstrap required, opened a new log
I20260927 18:36:35.297981 18973 ts_tablet_manager.cc:1397] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:36:35.298992 18972 tablet_bootstrap.cc:492] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Bootstrap starting.
I20260927 18:36:35.299642 18972 tablet_bootstrap.cc:654] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:35.300004 18972 log.cc:824] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:35.300235 18973 raft_consensus.cc:348] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.300462 18973 raft_consensus.cc:374] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:35.300568 18973 raft_consensus.cc:729] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f21dca85d54f8497d48d91c1716645, State: Initialized, Role: FOLLOWER
I20260927 18:36:35.300765 18973 consensus_queue.cc:260] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.301095 18973 ts_tablet_manager.cc:1428] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:35.301249 18805 heartbeater.cc:503] Master 127.5.151.254:41639 was elected leader, sending a full tablet report...
I20260927 18:36:35.302471 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.285792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:40785 (local address 127.0.0.1:53374)
0927 18:36:35.285953 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:35.285995 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:35.286059 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:35.298825 (+ 12766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:35.298826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:35.298828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:35.298864 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.298866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.300308 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.300309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.301340 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.301342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.302288 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.302289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.302346 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.302350 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:36:35.302376 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 18:36:35.302409 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260927 18:36:35.302847 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.286028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.5.151.193:40785)
0927 18:36:35.298656 (+ 12628us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:35.298662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:35.298673 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:36:35.298707 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:35.298709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:35.298724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:35.298813 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:35.298957 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.300220 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.301466 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.302186 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.302655 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.302716 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:35.302740 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 18:36:35.302777 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12589,"thread_start_us":45,"threads_started":1}
I20260927 18:36:35.303378 18605 tablet_service.cc:1467] Processing CreateTablet for tablet 15311bdaaf564bf88ac8703f16b57861 (DEFAULT_TABLE table=test-table [id=be39023a9b5d46e2ab0a59295a483d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:35.303686 18605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15311bdaaf564bf88ac8703f16b57861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:35.307134 18977 tablet_bootstrap.cc:492] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3: Bootstrap starting.
I20260927 18:36:35.307227 18972 tablet_bootstrap.cc:492] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: No bootstrap required, opened a new log
I20260927 18:36:35.307288 18972 ts_tablet_manager.cc:1397] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 18:36:35.307832 18977 tablet_bootstrap.cc:654] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:35.308138 18977 log.cc:824] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:35.308888 18977 tablet_bootstrap.cc:492] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3: No bootstrap required, opened a new log
I20260927 18:36:35.308943 18977 ts_tablet_manager.cc:1397] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
W20260927 18:36:35.309449 18806 tablet.cc:2369] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:36:35.309892 18972 raft_consensus.cc:348] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.310067 18972 raft_consensus.cc:374] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:35.310099 18972 raft_consensus.cc:729] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e47cf1202a04ada8cbfa66e81557c15, State: Initialized, Role: FOLLOWER
I20260927 18:36:35.310207 18972 consensus_queue.cc:260] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.310441 18972 ts_tablet_manager.cc:1428] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:36:35.310491 18939 heartbeater.cc:503] Master 127.5.151.254:41639 was elected leader, sending a full tablet report...
I20260927 18:36:35.311779 18977 raft_consensus.cc:348] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.311977 18977 raft_consensus.cc:374] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:35.312034 18977 raft_consensus.cc:729] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141c1ed3bbb2430c958a840135d8d9c3, State: Initialized, Role: FOLLOWER
I20260927 18:36:35.312156 18977 consensus_queue.cc:260] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.312430 18671 heartbeater.cc:503] Master 127.5.151.254:41639 was elected leader, sending a full tablet report...
I20260927 18:36:35.312414 18977 ts_tablet_manager.cc:1428] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260927 18:36:35.385015 18672 tablet.cc:2369] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:36:35 dist-test-slave-19nl krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534194, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for HTTP/127.5.151.212@KRBTEST.COM
W20260927 18:36:35.494181 18940 tablet.cc:2369] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:36:35.536147 18979 raft_consensus.cc:482] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:35.536281 18979 raft_consensus.cc:504] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.536660 18979 leader_election.cc:288] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 141c1ed3bbb2430c958a840135d8d9c3 (127.5.151.193:40785), e2f21dca85d54f8497d48d91c1716645 (127.5.151.194:41639)
I20260927 18:36:35.543368 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.537403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:51515 (local address 127.5.151.193:40785)
0927 18:36:35.537461 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:35.537463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:35.539903 (+ 2440us) server_negotiation.cc:407] Connection header received
0927 18:36:35.540011 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:35.540013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:35.540022 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:35.540048 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:35.540267 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.540993 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.542420 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.543003 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.543043 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.543052 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:35.543258 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 18:36:35.543297 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 18:36:35.543808 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.536919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:40785 (local address 127.5.151.195:51515)
0927 18:36:35.539828 (+ 2909us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:35.539841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:35.539931 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:35.540112 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:35.540113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:35.540115 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:35.540186 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.540188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.541086 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.541087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.542319 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.542322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.543121 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.543122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.543184 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.543188 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:36:35.543654 (+ 466us) client_negotiation.cc:240] Negotiation successful
0927 18:36:35.543707 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2888,"thread_start_us":90,"threads_started":1}
I20260927 18:36:35.544258 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15311bdaaf564bf88ac8703f16b57861" candidate_uuid: "1e47cf1202a04ada8cbfa66e81557c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141c1ed3bbb2430c958a840135d8d9c3" is_pre_election: true
I20260927 18:36:35.544412 18625 raft_consensus.cc:2454] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e47cf1202a04ada8cbfa66e81557c15 in term 0.
I20260927 18:36:35.544687 18827 leader_election.cc:302] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141c1ed3bbb2430c958a840135d8d9c3, 1e47cf1202a04ada8cbfa66e81557c15; no voters:
I20260927 18:36:35.544804 18979 raft_consensus.cc:2781] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:36:35.544852 18979 raft_consensus.cc:482] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:36:35.544898 18979 raft_consensus.cc:3037] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:35.545822 18979 raft_consensus.cc:504] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.545888 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.537456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:56689 (local address 127.5.151.194:41639)
0927 18:36:35.537511 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:35.537513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:35.542671 (+ 5158us) server_negotiation.cc:407] Connection header received
0927 18:36:35.542764 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:35.542765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:35.542774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:35.542802 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:35.542978 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.543656 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.544899 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.545437 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.545507 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.545515 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:35.545769 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 18:36:35.545818 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:36:35.545955 18979 leader_election.cc:288] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [CANDIDATE]: Term 1 election: Requested vote from peers 141c1ed3bbb2430c958a840135d8d9c3 (127.5.151.193:40785), e2f21dca85d54f8497d48d91c1716645 (127.5.151.194:41639)
I20260927 18:36:35.546070 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.537115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:41639 (local address 127.5.151.195:56689)
0927 18:36:35.542608 (+ 5493us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:35.542618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:35.542697 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:35.542857 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:35.542858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:35.542859 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:35.542910 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.542912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.543747 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.543748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.544783 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:35.544786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.545575 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.545575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:35.545680 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.545684 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:36:35.545950 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 18:36:35.545988 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5476,"thread_start_us":75,"threads_started":1}
I20260927 18:36:35.546414 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15311bdaaf564bf88ac8703f16b57861" candidate_uuid: "1e47cf1202a04ada8cbfa66e81557c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f21dca85d54f8497d48d91c1716645" is_pre_election: true
I20260927 18:36:35.546572 18759 raft_consensus.cc:2454] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e47cf1202a04ada8cbfa66e81557c15 in term 0.
I20260927 18:36:35.547284 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15311bdaaf564bf88ac8703f16b57861" candidate_uuid: "1e47cf1202a04ada8cbfa66e81557c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f21dca85d54f8497d48d91c1716645"
I20260927 18:36:35.547361 18759 raft_consensus.cc:3037] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:35.548127 18759 raft_consensus.cc:2454] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e47cf1202a04ada8cbfa66e81557c15 in term 1.
I20260927 18:36:35.548331 18828 leader_election.cc:302] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e47cf1202a04ada8cbfa66e81557c15, e2f21dca85d54f8497d48d91c1716645; no voters:
I20260927 18:36:35.548483 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15311bdaaf564bf88ac8703f16b57861" candidate_uuid: "1e47cf1202a04ada8cbfa66e81557c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141c1ed3bbb2430c958a840135d8d9c3"
I20260927 18:36:35.548550 18625 raft_consensus.cc:3037] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:35.549300 18625 raft_consensus.cc:2454] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e47cf1202a04ada8cbfa66e81557c15 in term 1.
I20260927 18:36:35.549532 18979 raft_consensus.cc:2781] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:35.549698 18979 raft_consensus.cc:686] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 1 LEADER]: Becoming Leader. State: Replica: 1e47cf1202a04ada8cbfa66e81557c15, State: Running, Role: LEADER
I20260927 18:36:35.549768 18979 consensus_queue.cc:237] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } }
I20260927 18:36:35.550468 18473 catalog_manager.cc:5495] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e47cf1202a04ada8cbfa66e81557c15 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "1e47cf1202a04ada8cbfa66e81557c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e47cf1202a04ada8cbfa66e81557c15" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20260927 18:36:35.576354 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.573068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.5.151.195:43801)
0927 18:36:35.573102 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:35.573104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:35.573110 (+ 6us) server_negotiation.cc:407] Connection header received
0927 18:36:35.573144 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:35.573144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:35.573159 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:35.573233 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:36:35.573392 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.574105 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.575345 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:35.575644 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:35.575702 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:35.575883 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:36:35.576011 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:36:35.576077 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:35.576272 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 18:36:35.576288 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260927 18:36:35.578313 18759 raft_consensus.cc:1261] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 1 FOLLOWER]: Refusing update from remote peer 1e47cf1202a04ada8cbfa66e81557c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:35.578683 18985 consensus_queue.cc:1035] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f21dca85d54f8497d48d91c1716645" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:35.580188 18625 raft_consensus.cc:1261] T 15311bdaaf564bf88ac8703f16b57861 P 141c1ed3bbb2430c958a840135d8d9c3 [term 1 FOLLOWER]: Refusing update from remote peer 1e47cf1202a04ada8cbfa66e81557c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:35.580466 18985 consensus_queue.cc:1035] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141c1ed3bbb2430c958a840135d8d9c3" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:35.583860 18988 mvcc.cc:205] Tried to move back new op lower bound from 7334028065086091264 to 7334028064972230656. Current Snapshot: MvccSnapshot[applied={T|T < 7334028065086091264}]
I20260927 18:36:35.583999 18989 mvcc.cc:205] Tried to move back new op lower bound from 7334028065086091264 to 7334028064972230656. Current Snapshot: MvccSnapshot[applied={T|T < 7334028065086091264}]
I20260927 18:36:35.586162 18991 mvcc.cc:205] Tried to move back new op lower bound from 7334028065086091264 to 7334028064972230656. Current Snapshot: MvccSnapshot[applied={T|T < 7334028065086091264}]
I20260927 18:36:35.588485 18473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:35.588562 18473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:35.589215 18473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f919aee429e4e6984219cd1a3968ff7: Sending DeleteTablet for 3 replicas of tablet 15311bdaaf564bf88ac8703f16b57861
I20260927 18:36:35.589500 18605 tablet_service.cc:1514] Processing DeleteTablet for tablet 15311bdaaf564bf88ac8703f16b57861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:35 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:53374
I20260927 18:36:35.589913 18873 tablet_service.cc:1514] Processing DeleteTablet for tablet 15311bdaaf564bf88ac8703f16b57861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:35 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47512
I20260927 18:36:35.590104 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 15311bdaaf564bf88ac8703f16b57861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:35 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:33842
I20260927 18:36:35.590654 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 18512
I20260927 18:36:35.592355 18998 tablet_replica.cc:330] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: stopping tablet replica
I20260927 18:36:35.592480 18999 tablet_replica.cc:330] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: stopping tablet replica
I20260927 18:36:35.592634 18998 raft_consensus.cc:2229] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:36:35.592764 18999 raft_consensus.cc:2229] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:35.592935 18998 raft_consensus.cc:2258] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:35.593076 18999 raft_consensus.cc:2258] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:35.594421 18999 ts_tablet_manager.cc:1905] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:35.594725 18998 ts_tablet_manager.cc:1905] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:35.597570 18999 ts_tablet_manager.cc:1918] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:35.600718 18998 ts_tablet_manager.cc:1918] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:35.600849 18999 log.cc:1196] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/15311bdaaf564bf88ac8703f16b57861
I20260927 18:36:35.600942 18998 log.cc:1196] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/15311bdaaf564bf88ac8703f16b57861
I20260927 18:36:35.601274 18999 ts_tablet_manager.cc:1939] T 15311bdaaf564bf88ac8703f16b57861 P e2f21dca85d54f8497d48d91c1716645: Deleting consensus metadata
I20260927 18:36:35.601558 18998 ts_tablet_manager.cc:1939] T 15311bdaaf564bf88ac8703f16b57861 P 1e47cf1202a04ada8cbfa66e81557c15: Deleting consensus metadata
I20260927 18:36:35.602406 18459 catalog_manager.cc:4841] TS 1e47cf1202a04ada8cbfa66e81557c15 (127.5.151.195:43801): tablet 15311bdaaf564bf88ac8703f16b57861 (table test-table [id=be39023a9b5d46e2ab0a59295a483d1d]) successfully deleted
I20260927 18:36:35.602895 18459 catalog_manager.cc:4841] TS e2f21dca85d54f8497d48d91c1716645 (127.5.151.194:41639): tablet 15311bdaaf564bf88ac8703f16b57861 (table test-table [id=be39023a9b5d46e2ab0a59295a483d1d]) successfully deleted
W20260927 18:36:35.607487 18458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:40785 (error 108)
I20260927 18:36:35.608367 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 18675
I20260927 18:36:35.614413 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 18807
I20260927 18:36:35.620565 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:35.620420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:40785 (local address 127.0.0.1:53382)
0927 18:36:35.620474 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:35.620531 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 18:36:35.620707 18458 catalog_manager.cc:4584] TS 141c1ed3bbb2430c958a840135d8d9c3 (127.5.151.193:40785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15311bdaaf564bf88ac8703f16b57861: Network error: Client connection negotiation failed: client connection to 127.5.151.193:40785: connect: Connection refused (error 111)
I20260927 18:36:35.621059 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 18440
I20260927 18:36:35.628082 5727 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:36:35.962025 5727 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:36:35.962139 5727 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:36:36.305701 5727 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:36:36.305 UTC [17507] LOG: received smart shutdown request
2026-09-27 18:36:36.309 UTC [17507] LOG: background worker "logical replication launcher" (PID 17516) exited with exit code 1
2026-09-27 18:36:36.309 UTC [17511] LOG: shutting down
2026-09-27 18:36:36.446 UTC [17507] LOG: database system is shut down
2026-09-27T18:36:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45084 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:36:36 dist-test-slave-19nl krb5kdc[19009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:36:36 dist-test-slave-19nl krb5kdc[19009](info): set up 2 sockets
Sep 27 18:36:36 dist-test-slave-19nl krb5kdc[19009](info): commencing operation
krb5kdc: starting...
W20260927 18:36:37.554198 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:36:37 dist-test-slave-19nl krb5kdc[19009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:36:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:36:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:36:37.659354 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45827
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:45827,127.5.151.253:38173,127.5.151.252:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:37.735502 19022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:37.735723 19022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:37.735755 19022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:37.736799 19022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:37.736857 19022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:37.736879 19022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:37.736903 19022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:37.736931 19022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:37.738287 19022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:45827,127.5.151.253:38173,127.5.151.252:39711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:37.738528 19022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:37.738744 19022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:37.742889 19027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:37.742877 19030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:37.742852 19028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:37.743157 19022 server_base.cc:1034] running on GCE node
I20260927 18:36:37.743382 19022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:37.743690 19022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:37.744825 19022 hybrid_clock.cc:648] HybridClock initialized: now 1790534197744810 us; error 30 us; skew 500 ppm
Sep 27 18:36:37 dist-test-slave-19nl krb5kdc[19009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:37.747175 19022 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:37.747517 19022 webserver.cc:460] Webserver started at http://127.5.151.254:41429/ using document root <none> and password file <none>
I20260927 18:36:37.747676 19022 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:37.747712 19022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:37.747792 19022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:37.748693 19022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "2ed8deffb3754a33995e58bca05dd674"
format_stamp: "Formatted at 2026-09-27 18:36:37 on dist-test-slave-19nl"
server_key: "dd963d8eacc3923cea4800073cef14aa"
server_key_iv: "bc6465a3733143c1e57155baa284b9cb"
server_key_version: "encryptionkey@0"
I20260927 18:36:37.748965 19022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "2ed8deffb3754a33995e58bca05dd674"
format_stamp: "Formatted at 2026-09-27 18:36:37 on dist-test-slave-19nl"
server_key: "dd963d8eacc3923cea4800073cef14aa"
server_key_iv: "bc6465a3733143c1e57155baa284b9cb"
server_key_version: "encryptionkey@0"
I20260927 18:36:37.750094 19022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:36:37.750701 19037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:37.750859 19022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:37.750946 19022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "2ed8deffb3754a33995e58bca05dd674"
format_stamp: "Formatted at 2026-09-27 18:36:37 on dist-test-slave-19nl"
server_key: "dd963d8eacc3923cea4800073cef14aa"
server_key_iv: "bc6465a3733143c1e57155baa284b9cb"
server_key_version: "encryptionkey@0"
I20260927 18:36:37.751025 19022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:37.758633 19022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:37.760691 19022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:37.760828 19022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:37.764434 19022 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:45827
I20260927 18:36:37.764482 19089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:45827 every 8 connection(s)
I20260927 18:36:37.764814 19022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:37.765581 19090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:37.767050 19090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:36:37 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:37.773114 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19022
I20260927 18:36:37.773248 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:36:37.773454 5727 external_mini_cluster.cc:1428] Setting key f7bc17a486e9b816c0622a2d16c53e80
I20260927 18:36:37.776183 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.767757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.5.151.254:45827)
0927 18:36:37.767930 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:37.767940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:37.768071 (+ 131us) server_negotiation.cc:407] Connection header received
0927 18:36:37.768586 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:37.768588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:37.768603 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:37.768683 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:37.769147 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.769936 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.770737 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.771013 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.771086 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.772819 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:37.772825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:37.772825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:37.772831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:37.774664 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.775014 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.775017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.775017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.775096 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.775311 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.775311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.775312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.775504 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:37.775588 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:37.775777 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:36:37.775912 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260927 18:36:37.776190 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.767747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45827 (local address 127.0.0.1:54164)
0927 18:36:37.767926 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.767962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:37.768454 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:37.768705 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:37.768707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:37.768923 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:37.769070 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.769073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.770011 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.770012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.770675 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.770678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.771078 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.771079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.771128 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.771499 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:37.771507 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:37.772738 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:37.774736 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.774742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.774743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.774920 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.775162 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.775163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.775163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.775247 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.775569 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:37.775574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:37.775629 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 18:36:37.775753 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 18:36:37.775974 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":93,"thread_start_us":99,"threads_started":1}
I20260927 18:36:37.776914 19090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:36:37.777251 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.777125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36556)
0927 18:36:37.777173 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.777220 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:36:37.777393 19039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
I20260927 18:36:37.777835 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.777729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36564)
0927 18:36:37.777784 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.777813 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 18:36:37.777968 19090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:38173: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.151.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab.
Entry for principal kudu/127.5.151.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab.
I20260927 18:36:37.795540 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:38173
--webserver_interface=127.5.151.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab
--principal=kudu/127.5.151.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:45827,127.5.151.253:38173,127.5.151.252:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:37.827154 19090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } attempt: 1
I20260927 18:36:37.827555 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.827434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36580)
0927 18:36:37.827492 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.827528 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:36:37.827893 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.827784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36592)
0927 18:36:37.827843 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.827866 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 18:36:37.827978 19090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:38173: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
W20260927 18:36:37.869576 19097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:37.869745 19097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:37.869764 19097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:37.870859 19097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:37.870895 19097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:37.870908 19097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:37.870949 19097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:37.870967 19097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:37.872371 19097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:45827,127.5.151.253:38173,127.5.151.252:39711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab
--principal=kudu/127.5.151.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.151.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:37.872597 19097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:37.872802 19097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:37.876984 19105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:37.877002 19102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:37.877100 19103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:37.877214 19097 server_base.cc:1034] running on GCE node
I20260927 18:36:37.877408 19097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:37.877641 19097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:37.878790 19097 hybrid_clock.cc:648] HybridClock initialized: now 1790534197878767 us; error 31 us; skew 500 ppm
Sep 27 18:36:37 dist-test-slave-19nl krb5kdc[19009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:37.880900 19097 init.cc:377] Logged in from keytab as kudu/127.5.151.253@KRBTEST.COM (short username kudu)
I20260927 18:36:37.881316 19097 webserver.cc:460] Webserver started at http://127.5.151.253:33773/ using document root <none> and password file <none>
I20260927 18:36:37.881527 19097 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:37.881587 19097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:37.881709 19097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:37.882712 19097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data/instance:
uuid: "6947a1e32aae437583c19285dba505d1"
format_stamp: "Formatted at 2026-09-27 18:36:37 on dist-test-slave-19nl"
server_key: "407539ad68feda8fe8459932830ce0e7"
server_key_iv: "6e5852bc697843aca538f473fff2d21b"
server_key_version: "encryptionkey@0"
I20260927 18:36:37.883051 19097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/wal/instance:
uuid: "6947a1e32aae437583c19285dba505d1"
format_stamp: "Formatted at 2026-09-27 18:36:37 on dist-test-slave-19nl"
server_key: "407539ad68feda8fe8459932830ce0e7"
server_key_iv: "6e5852bc697843aca538f473fff2d21b"
server_key_version: "encryptionkey@0"
I20260927 18:36:37.884222 19097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:36:37.884851 19112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:37.885010 19097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:36:37.885095 19097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/wal
uuid: "6947a1e32aae437583c19285dba505d1"
format_stamp: "Formatted at 2026-09-27 18:36:37 on dist-test-slave-19nl"
server_key: "407539ad68feda8fe8459932830ce0e7"
server_key_iv: "6e5852bc697843aca538f473fff2d21b"
server_key_version: "encryptionkey@0"
I20260927 18:36:37.885170 19097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:37.896139 19090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } attempt: 2
I20260927 18:36:37.896513 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.896396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36598)
0927 18:36:37.896450 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.896483 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 18:36:37.896880 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.896757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36610)
0927 18:36:37.896828 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.896858 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:36:37.897018 19090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:38173: Network error: Client connection negotiation failed: client connection to 127.5.151.253:38173: connect: Connection refused (error 111)
I20260927 18:36:37.906075 19097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:37.907735 19097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:37.907835 19097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:37.911600 19097 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.253:38173
I20260927 18:36:37.911645 19164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.253:38173 every 8 connection(s)
I20260927 18:36:37.911931 19097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
I20260927 18:36:37.912767 19165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:37.914193 19165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:36:37 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:37.919307 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19097
I20260927 18:36:37.919414 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-1/wal/instance
I20260927 18:36:37.919625 5727 external_mini_cluster.cc:1428] Setting key 6a5f138742d4f0a5c26fb318a926cacd
I20260927 18:36:37.922703 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.914869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.5.151.254:45827)
0927 18:36:37.914903 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:37.914905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:37.915083 (+ 178us) server_negotiation.cc:407] Connection header received
0927 18:36:37.915474 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:37.915475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:37.915485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:37.915540 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:37.916084 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.916536 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.917439 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.917625 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.917673 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.919717 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:37.919718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:37.919727 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:37.919733 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:37.921370 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.921727 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.921728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.921729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.921787 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.922067 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.922068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.922069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.922177 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:37.922250 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:37.922522 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 18:36:37.922617 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:36:37.922904 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.914834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45827 (local address 127.0.0.1:54180)
0927 18:36:37.914967 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.915003 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:37.915402 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:37.915587 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:37.915589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:37.915848 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:37.916020 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.916023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.916610 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.916610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.917367 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.917369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.917693 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.917694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.917788 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.918173 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:37.918181 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:37.919605 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:37.921461 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.921467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.921468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.921656 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.921864 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.921865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.921865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.921978 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.922243 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:37.922249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:37.922381 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 18:36:37.922499 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:36:37.922675 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260927 18:36:37.923384 19165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.151.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.252@KRBTEST.COM" created.
Sep 27 18:36:37 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:36:37.932524 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.923619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36622)
0927 18:36:37.923662 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.923671 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:37.923706 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:37.923980 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:37.923981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:37.924084 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:37.924126 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.924127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.925437 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.925438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.925902 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.925903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.926481 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.926481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.926528 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.926925 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:37.926929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:37.929906 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:37.931574 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.931575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.931575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.931762 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.931979 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.931979 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.931979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.932027 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.932318 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:37.932319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:37.932349 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:36:37.932398 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:36:37.932459 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 18:36:37.932668 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.923669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.5.151.253:38173)
0927 18:36:37.923825 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:37.923829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:37.923852 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:36:37.923875 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:37.923878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:37.923892 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:37.923974 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:37.924194 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.925346 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.925974 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.926406 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.926488 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.929991 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:37.929998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:37.929998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:37.930004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:37.931502 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.931848 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.931850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.931851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.931913 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.932086 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.932087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.932087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.932256 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:37.932335 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:37.932446 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 18:36:37.932539 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260927 18:36:37.933238 19165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:36:37.933609 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.933450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35148)
0927 18:36:37.933529 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.933580 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 18:36:37.934121 19115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
I20260927 18:36:37.934607 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.934475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35150)
0927 18:36:37.934532 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.934583 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:36:37.934792 19165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:39711: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab.
Entry for principal kudu/127.5.151.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab.
I20260927 18:36:37.943364 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:39711
--webserver_interface=127.5.151.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab
--principal=kudu/127.5.151.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:45827,127.5.151.253:38173,127.5.151.252:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:37.984017 19165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } attempt: 1
I20260927 18:36:37.985056 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.984406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35164)
0927 18:36:37.984945 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.985000 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482}
I20260927 18:36:37.985541 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.985326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35174)
0927 18:36:37.985476 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.985504 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
W20260927 18:36:37.985625 19165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:39711: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
I20260927 18:36:37.988178 19090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } attempt: 3
Sep 27 18:36:37 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:36:37.996073 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.988459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36624)
0927 18:36:37.988526 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.988539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:37.988705 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:37.989279 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:37.989280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:37.989482 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:37.989563 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.989565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.990579 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.990580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.991079 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:37.991081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.991579 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.991580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:37.991647 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.992123 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:37.992129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:37.993316 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:37.995081 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.995081 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.995082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.995241 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.995451 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:37.995451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:37.995452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:37.995503 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:37.995792 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:37.995793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:37.995840 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 18:36:37.995909 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:36:37.996004 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 18:36:37.996119 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.988668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.5.151.253:38173)
0927 18:36:37.989146 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:37.989148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:37.989156 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:36:37.989183 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:37.989184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:37.989196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:37.989632 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:37.989667 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.990487 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.991158 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:37.991504 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:37.991574 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:37.993401 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:37.993402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:37.993403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:37.993416 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:37.995007 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.995313 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.995314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.995314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.995377 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:37.995561 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:37.995562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:37.995562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:37.995694 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:37.995763 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:37.995951 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:36:37.996034 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453}
I20260927 18:36:37.996539 19090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:36:37.996912 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.996800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35176)
0927 18:36:37.996855 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.996887 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:36:37.997294 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:37.997216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35178)
0927 18:36:37.997248 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:37.997275 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 18:36:37.997411 19090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:39711: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
W20260927 18:36:38.024848 19172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:38.025147 19172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:38.025180 19172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:38.026633 19172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:38.026681 19172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:38.026695 19172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:38.026708 19172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:38.026749 19172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:38.028288 19172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:45827,127.5.151.253:38173,127.5.151.252:39711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab
--principal=kudu/127.5.151.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.151.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:38.028545 19172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:38.028565 19090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } attempt: 1
I20260927 18:36:38.028797 19172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:36:38.029022 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.028893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35190)
0927 18:36:38.028945 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.028989 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 18:36:38.029387 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.029295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35194)
0927 18:36:38.029342 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.029367 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 18:36:38.029507 19090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:39711: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
W20260927 18:36:38.033429 19179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:38.033460 19178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:38.033460 19181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:38.033753 19172 server_base.cc:1034] running on GCE node
I20260927 18:36:38.033944 19172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:38.034204 19172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:38.035337 19172 hybrid_clock.cc:648] HybridClock initialized: now 1790534198035319 us; error 30 us; skew 500 ppm
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534198, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:38.037642 19172 init.cc:377] Logged in from keytab as kudu/127.5.151.252@KRBTEST.COM (short username kudu)
I20260927 18:36:38.038074 19172 webserver.cc:460] Webserver started at http://127.5.151.252:44613/ using document root <none> and password file <none>
I20260927 18:36:38.038287 19172 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:38.038347 19172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:38.038486 19172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:38.039495 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data/instance:
uuid: "8c5a5ac258be4719832f4fab8f3e6f2e"
format_stamp: "Formatted at 2026-09-27 18:36:38 on dist-test-slave-19nl"
server_key: "7b7bafb1177643003eea61230df90602"
server_key_iv: "a870a8d894e8652145c1c1b68e5c48cb"
server_key_version: "encryptionkey@0"
I20260927 18:36:38.039846 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/wal/instance:
uuid: "8c5a5ac258be4719832f4fab8f3e6f2e"
format_stamp: "Formatted at 2026-09-27 18:36:38 on dist-test-slave-19nl"
server_key: "7b7bafb1177643003eea61230df90602"
server_key_iv: "a870a8d894e8652145c1c1b68e5c48cb"
server_key_version: "encryptionkey@0"
I20260927 18:36:38.041126 19172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:36:38.041771 19188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:38.041932 19172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:38.042023 19172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/wal
uuid: "8c5a5ac258be4719832f4fab8f3e6f2e"
format_stamp: "Formatted at 2026-09-27 18:36:38 on dist-test-slave-19nl"
server_key: "7b7bafb1177643003eea61230df90602"
server_key_iv: "a870a8d894e8652145c1c1b68e5c48cb"
server_key_version: "encryptionkey@0"
I20260927 18:36:38.042099 19172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:38.050173 19172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:38.051887 19172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:38.052042 19172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:38.053730 19165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } attempt: 2
I20260927 18:36:38.054177 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.053992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35206)
0927 18:36:38.054064 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.054127 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 18:36:38.054541 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.054423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35214)
0927 18:36:38.054471 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.054513 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:36:38.054651 19165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:39711: Network error: Client connection negotiation failed: client connection to 127.5.151.252:39711: connect: Connection refused (error 111)
I20260927 18:36:38.056169 19172 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.252:39711
I20260927 18:36:38.056198 19240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.252:39711 every 8 connection(s)
I20260927 18:36:38.056515 19172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
I20260927 18:36:38.057284 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:38.057780 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19172
I20260927 18:36:38.057861 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534055081918-5727-0/minicluster-data/master-2/wal/instance
I20260927 18:36:38.058069 5727 external_mini_cluster.cc:1428] Setting key 5151859b3d5c692a14c04b0927d32c28
I20260927 18:36:38.059290 19241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534198, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:36:38.067425 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.059950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.5.151.254:45827)
0927 18:36:38.059997 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.060000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.060121 (+ 121us) server_negotiation.cc:407] Connection header received
0927 18:36:38.060198 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.060199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.060208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.060259 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.060712 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.061178 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.062016 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.062210 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.062289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.064314 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.064315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.064315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.064321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.065920 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.066354 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.066355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.066355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.066401 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.066680 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.066681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.066681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.066832 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.066898 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.067267 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.067329 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 18:36:38.067607 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.059882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:45827 (local address 127.0.0.1:54182)
0927 18:36:38.060019 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.060056 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:38.060139 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:38.060276 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:38.060279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:38.060521 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:38.060607 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.060611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.061257 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.061258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.061928 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.061930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.062300 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.062301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.062378 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.062833 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:38.062840 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:38.064236 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:38.065988 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.065992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.066006 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.066283 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.066475 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.066476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.066476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.066625 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.066903 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:38.066908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:38.067083 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 18:36:38.067234 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:36:38.067383 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260927 18:36:38.067991 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.058576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.5.151.252:39711)
0927 18:36:38.058759 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.058768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.058803 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:36:38.059294 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.059301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.059356 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.059492 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.060194 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.061002 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.061715 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.062054 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.062159 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.064788 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.064795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.064795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.064802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.066532 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.066965 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.066968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.066968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.067026 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.067196 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.067197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.067197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.067339 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.067459 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.067808 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.067888 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260927 18:36:38.068113 19241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534198, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:38.075181 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.068340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:38173 (local address 127.0.0.1:36636)
0927 18:36:38.068384 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.068394 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:38.068432 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:38.068578 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:38.068579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:38.068694 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:38.068720 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.068721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.069373 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.069374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.069834 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.069835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.070213 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.070214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.070267 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.070673 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:38.070678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:38.071974 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:38.074190 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.074191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.074191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.074348 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.074566 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.074567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.074568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.074645 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.074902 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:38.074903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:38.074942 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:36:38.075007 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:36:38.075094 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 18:36:38.075186 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.068416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.5.151.253:38173)
0927 18:36:38.068459 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.068460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.068468 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:36:38.068510 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.068515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.068524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.068572 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.068799 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.069288 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.069916 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.070131 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.070213 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.072063 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.072064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.072064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.072071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.074096 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.074431 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.074432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.074433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.074493 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.074724 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.074725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.074725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.074835 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.074909 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.075037 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.075105 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:36:38.075695 19241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:36:38.076370 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.068886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.5.151.254:45827)
0927 18:36:38.068933 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.068934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.068943 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:36:38.068962 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.068963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.068970 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.069022 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.069467 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.069887 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.070545 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.070760 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.070839 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.072995 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.072996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.072997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.073004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.074409 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.074794 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.074795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.074796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.074850 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.075045 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.075046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.075046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.075146 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.076219 (+ 1073us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.076252 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.076313 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534198, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:36:38.082679 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.075991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.5.151.252:39711)
0927 18:36:38.076025 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.076027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.076034 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:36:38.076053 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.076053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.076061 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.076111 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.076328 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.076922 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.077897 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.078202 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.078271 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.080112 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.080113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.080114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.080119 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.081438 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.081815 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.081816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.081816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.081869 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.082084 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.082085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.082085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.082201 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.082269 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.082412 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.082530 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":20}
I20260927 18:36:38.082693 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.075926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35226)
0927 18:36:38.075968 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.075977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:38.076011 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:38.076114 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:38.076114 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:38.076237 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:38.076257 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.076259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.077157 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.077158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.077828 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.077830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.078282 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.078282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.078332 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.078729 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:38.078734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:38.080041 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:38.081522 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.081523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.081523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.081754 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.081952 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.081953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.081953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.082008 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.082274 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:38.082275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:38.082309 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:36:38.082366 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:36:38.082483 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 18:36:38.083968 19241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e: Bootstrap starting.
I20260927 18:36:38.084297 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.076944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.5.151.253:38173)
0927 18:36:38.076986 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.076988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.077022 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:36:38.077038 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.077039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.077046 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.077088 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.077612 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.078191 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.079167 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.079437 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.079517 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.081727 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.081728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.081728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.081735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.083269 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.083584 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.083585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.083585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.083635 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.083831 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.083831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.083832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.083936 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.084106 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.084140 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.084209 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:36:38.085044 19241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:38.085373 19241 log.cc:824] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:38.085687 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:36:38.086154 19241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e: No bootstrap required, opened a new log
I20260927 18:36:38.087963 19241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.088107 19241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:38.088153 19241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5a5ac258be4719832f4fab8f3e6f2e, State: Initialized, Role: FOLLOWER
I20260927 18:36:38.088274 19241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.088559 19241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:38.088510 19254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.088668 19254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:36:38.090512 19172 master_runner.cc:186] Error getting master registration for 127.5.151.254:45827: OK, instance_id { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" instance_seqno: 1790534197763455 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:36:38.093554 19172 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 18:36:38.093678 19265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:36:38.093724 19265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:36:38.093991 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.087034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.5.151.254:45827)
0927 18:36:38.087072 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.087074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.087354 (+ 280us) server_negotiation.cc:407] Connection header received
0927 18:36:38.087425 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.087426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.087435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.087510 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.088084 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.088721 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.089441 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.089678 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.089753 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.091259 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.091260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.091261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.091267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.092801 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.093166 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.093167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.093168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.093218 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.093399 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.093399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.093400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.093522 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.093610 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.093833 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.093905 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:36:38.097090 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.087158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.5.151.252:39711)
0927 18:36:38.087238 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.087239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.089969 (+ 2730us) server_negotiation.cc:407] Connection header received
0927 18:36:38.090364 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.090365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.090378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.090430 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.091144 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.091818 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.092743 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.093018 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.093052 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.094635 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.094636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.094637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.094643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.095923 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.096244 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.096246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.096246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.096296 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.096511 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.096513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.096513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.096666 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.096766 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.096923 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.097010 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:36:38.098270 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.087192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.5.151.253:38173)
0927 18:36:38.087230 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.087232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.090461 (+ 3229us) server_negotiation.cc:407] Connection header received
0927 18:36:38.090551 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.090552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.090584 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.090613 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.091196 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.092315 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.093195 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.093720 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.093762 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.095548 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.095550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.095550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.095557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.097267 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.097623 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.097624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.097624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.097694 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.097878 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.097878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.097878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.097958 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.098035 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.098163 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.098216 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:36:38.104660 19090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } attempt: 2
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:36:38.111253 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.104953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.5.151.252:39711)
0927 18:36:38.105030 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.105033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.105048 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:36:38.105075 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.105076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.105083 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.105117 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.105372 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.105774 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.106700 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.106860 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.106909 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.108814 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.108815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.108815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.108822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.110232 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.110633 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.110633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.110634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.110679 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.110865 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.110866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.110866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.110955 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.111011 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.111146 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.111197 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:36:38.111311 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.104896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35244)
0927 18:36:38.104951 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.104963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:38.105027 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:38.105121 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:38.105122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:38.105272 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:38.105311 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.105313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.105842 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.105844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.106632 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.106635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.106926 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.106927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.107025 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.107420 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:38.107425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:38.108713 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:38.110307 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.110308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.110308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.110517 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.110739 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.110740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.110740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.110805 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.111012 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:38.111013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:38.111058 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:36:38.111127 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:36:38.111217 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:36:38.112627 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: Bootstrap starting.
I20260927 18:36:38.113277 19090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:38.113569 19090 log.cc:824] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:38.114339 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: No bootstrap required, opened a new log
I20260927 18:36:38.116022 19090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.116148 19090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:38.116197 19090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed8deffb3754a33995e58bca05dd674, State: Initialized, Role: FOLLOWER
I20260927 18:36:38.116320 19090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.116628 19268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.116693 19090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:38.116704 19268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:36:38.118439 19279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:36:38.118510 19279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:36:38.119321 19022 master_runner.cc:186] Error getting master registration for 127.5.151.253:38173: OK, instance_id { permanent_uuid: "6947a1e32aae437583c19285dba505d1" instance_seqno: 1790534197910506 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:36:38.122025 19022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:36:38.135581 19254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:38.135702 19254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.135973 19254 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ed8deffb3754a33995e58bca05dd674 (127.5.151.254:45827), 6947a1e32aae437583c19285dba505d1 (127.5.151.253:38173)
I20260927 18:36:38.136228 19064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed8deffb3754a33995e58bca05dd674" is_pre_election: true
I20260927 18:36:38.136349 19064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5a5ac258be4719832f4fab8f3e6f2e in term 0.
I20260927 18:36:38.136559 19190 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ed8deffb3754a33995e58bca05dd674, 8c5a5ac258be4719832f4fab8f3e6f2e; no voters:
I20260927 18:36:38.136710 19254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:36:38.136778 19254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:36:38.136806 19254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:38.137717 19254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.137842 19254 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [CANDIDATE]: Term 1 election: Requested vote from peers 2ed8deffb3754a33995e58bca05dd674 (127.5.151.254:45827), 6947a1e32aae437583c19285dba505d1 (127.5.151.253:38173)
I20260927 18:36:38.138055 19064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed8deffb3754a33995e58bca05dd674"
I20260927 18:36:38.138127 19064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:38.139424 19064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5a5ac258be4719832f4fab8f3e6f2e in term 1.
I20260927 18:36:38.139710 19190 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ed8deffb3754a33995e58bca05dd674, 8c5a5ac258be4719832f4fab8f3e6f2e; no voters:
I20260927 18:36:38.139885 19254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:38.140092 19254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [term 1 LEADER]: Becoming Leader. State: Replica: 8c5a5ac258be4719832f4fab8f3e6f2e, State: Running, Role: LEADER
I20260927 18:36:38.140192 19254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.140605 19281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5a5ac258be4719832f4fab8f3e6f2e. Latest consensus state: current_term: 1 leader_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.140793 19281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:38.141276 19283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:38.141695 19283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:38.143494 19064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [term 1 FOLLOWER]: Refusing update from remote peer 8c5a5ac258be4719832f4fab8f3e6f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:38.143813 19281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:38.145170 19268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5a5ac258be4719832f4fab8f3e6f2e. Latest consensus state: current_term: 1 leader_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.145262 19268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:36:38.145812 19165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } attempt: 3
I20260927 18:36:38.146791 19281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.146881 19281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:38.147312 19268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.147411 19268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:36:38.147699 19284 mvcc.cc:205] Tried to move back new op lower bound from 7334028075593084928 to 7334028075582853120. Current Snapshot: MvccSnapshot[applied={T|T < 7334028075593084928}]
I20260927 18:36:38.148514 19285 mvcc.cc:205] Tried to move back new op lower bound from 7334028075593084928 to 7334028075582853120. Current Snapshot: MvccSnapshot[applied={T|T < 7334028075593084928}]
Sep 27 18:36:38 dist-test-slave-19nl krb5kdc[19009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:36:38.154520 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.146044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:39711 (local address 127.0.0.1:35250)
0927 18:36:38.146122 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:38.146133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:38.146176 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:38.146339 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:38.146340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:38.146543 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:38.146628 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.146630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.147558 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.147559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.148392 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:38.148395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.148857 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.148858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:38.148938 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.149450 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:38.149455 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:38.151393 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:38.153388 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.153389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.153389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.153614 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.153825 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:38.153825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:38.153826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:38.153910 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:38.154193 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:38.154195 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:38.154240 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:36:38.154308 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:36:38.154424 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:36:38.154896 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:38.146136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.5.151.252:39711)
0927 18:36:38.146174 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:38.146176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:38.146185 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:36:38.146227 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:38.146228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:38.146238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:38.146290 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:38.146730 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.147464 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.148471 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:38.148771 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:38.148840 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:38.151576 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:38.151577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:38.151578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:38.151585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:38.153309 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.153689 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.153690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.153690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.153749 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:38.153967 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:38.153968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:38.153969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:38.154105 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:38.154165 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:38.154388 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:36:38.154503 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:36:38.154976 19283 catalog_manager.cc:1347] Generated new cluster ID: 87716cfb5c874866949471ef95aa7c77
I20260927 18:36:38.155310 19283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:36:38.163888 19165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: Bootstrap starting.
I20260927 18:36:38.164598 19165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:38.164950 19165 log.cc:824] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:38.165993 19165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: No bootstrap required, opened a new log
I20260927 18:36:38.167943 19165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.168080 19165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:38.168125 19165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6947a1e32aae437583c19285dba505d1, State: Initialized, Role: FOLLOWER
I20260927 18:36:38.168242 19165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } }
I20260927 18:36:38.168586 19165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:38.168941 19138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6947a1e32aae437583c19285dba505d1"
I20260927 18:36:38.169046 19138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:38.169852 19138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5a5ac258be4719832f4fab8f3e6f2e in term 1.
I20260927 18:36:38.170197 19136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [term 1 FOLLOWER]: Refusing update from remote peer 8c5a5ac258be4719832f4fab8f3e6f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:38.170408 19139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6947a1e32aae437583c19285dba505d1" is_pre_election: true
I20260927 18:36:38.170475 19139 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8c5a5ac258be4719832f4fab8f3e6f2e in term 1. Re-sending same reply.
I20260927 18:36:38.170764 19281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:38.173193 19290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.175137 19290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:36:38.175231 19290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5a5ac258be4719832f4fab8f3e6f2e. Latest consensus state: current_term: 1 leader_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.175334 19290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:36:38.175398 19290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.176381 19290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:36:38.178982 19292 mvcc.cc:205] Tried to move back new op lower bound from 7334028075593084928 to 7334028075582853120. Current Snapshot: MvccSnapshot[applied={T|T < 7334028075593084928}]
W20260927 18:36:38.179891 19305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:36:38.179979 19305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:36:38.184451 19281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed8deffb3754a33995e58bca05dd674" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45827 } } peers { permanent_uuid: "6947a1e32aae437583c19285dba505d1" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 38173 } } peers { permanent_uuid: "8c5a5ac258be4719832f4fab8f3e6f2e" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 39711 } } }
I20260927 18:36:38.184552 19281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:38.186264 19283 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:36:38.187011 19283 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:38.202521 19283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c5a5ac258be4719832f4fab8f3e6f2e: Generated new TSK 0
I20260927 18:36:38.202739 19283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:36:39.118923 19279 catalog_manager.cc:1259] Loaded cluster ID: 87716cfb5c874866949471ef95aa7c77
I20260927 18:36:39.118999 19279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: loading cluster ID for follower catalog manager: success
I20260927 18:36:39.119796 19279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: acquiring CA information for follower catalog manager: success
I20260927 18:36:39.120097 19279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ed8deffb3754a33995e58bca05dd674: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:36:39.180420 19305 catalog_manager.cc:1259] Loaded cluster ID: 87716cfb5c874866949471ef95aa7c77
I20260927 18:36:39.180505 19305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: loading cluster ID for follower catalog manager: success
I20260927 18:36:39.181290 19305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: acquiring CA information for follower catalog manager: success
I20260927 18:36:39.181591 19305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6947a1e32aae437583c19285dba505d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:36:39.251798 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19022
I20260927 18:36:39.256866 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19097
I20260927 18:36:39.261441 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19172
2026-09-27T18:36:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2713 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:36:39 dist-test-slave-19nl krb5kdc[19315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:36:39 dist-test-slave-19nl krb5kdc[19315](info): set up 2 sockets
Sep 27 18:36:39 dist-test-slave-19nl krb5kdc[19315](info): commencing operation
krb5kdc: starting...
W20260927 18:36:40.254288 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:36:40 dist-test-slave-19nl krb5kdc[19315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:36:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:36:40.358717 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36611
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:36611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:40.434892 19328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:40.435118 19328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:40.435151 19328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:40.436223 19328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:40.436285 19328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:40.436308 19328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:40.436328 19328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:40.436359 19328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:40.437755 19328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:36611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:40.438010 19328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:40.438225 19328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:40.442835 19334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:40.442809 19336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:40.442811 19333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:40.443094 19328 server_base.cc:1034] running on GCE node
I20260927 18:36:40.443360 19328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:40.443653 19328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:40.444792 19328 hybrid_clock.cc:648] HybridClock initialized: now 1790534200444780 us; error 27 us; skew 500 ppm
Sep 27 18:36:40 dist-test-slave-19nl krb5kdc[19315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534200, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:40.450399 19328 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:40.450907 19328 webserver.cc:460] Webserver started at http://127.5.151.254:45039/ using document root <none> and password file <none>
I20260927 18:36:40.451155 19328 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:40.451220 19328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:40.451364 19328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:40.452382 19328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "993fe230d2264da3834e747131523981"
format_stamp: "Formatted at 2026-09-27 18:36:40 on dist-test-slave-19nl"
server_key: "05f928d9cf0e8f5b8ecd2f32c4279c73"
server_key_iv: "e7ffc58ea81770d6107803d0adf16135"
server_key_version: "encryptionkey@0"
I20260927 18:36:40.452739 19328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "993fe230d2264da3834e747131523981"
format_stamp: "Formatted at 2026-09-27 18:36:40 on dist-test-slave-19nl"
server_key: "05f928d9cf0e8f5b8ecd2f32c4279c73"
server_key_iv: "e7ffc58ea81770d6107803d0adf16135"
server_key_version: "encryptionkey@0"
I20260927 18:36:40.454032 19328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:36:40.454807 19343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:40.454979 19328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:36:40.455070 19328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "993fe230d2264da3834e747131523981"
format_stamp: "Formatted at 2026-09-27 18:36:40 on dist-test-slave-19nl"
server_key: "05f928d9cf0e8f5b8ecd2f32c4279c73"
server_key_iv: "e7ffc58ea81770d6107803d0adf16135"
server_key_version: "encryptionkey@0"
I20260927 18:36:40.455155 19328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:40.468446 19328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:40.470966 19328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:40.471131 19328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:40.475282 19328 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:36611
I20260927 18:36:40.475330 19395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:36611 every 8 connection(s)
I20260927 18:36:40.475689 19328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:40.476503 19396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:40.478773 19396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981: Bootstrap starting.
I20260927 18:36:40.479358 19396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:40.479637 19396 log.cc:824] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:40.480276 19396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981: No bootstrap required, opened a new log
I20260927 18:36:40.481796 19396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993fe230d2264da3834e747131523981" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36611 } }
I20260927 18:36:40.481925 19396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:40.481950 19396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993fe230d2264da3834e747131523981, State: Initialized, Role: FOLLOWER
I20260927 18:36:40.482146 19396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993fe230d2264da3834e747131523981" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36611 } }
I20260927 18:36:40.482224 19396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:40.482262 19396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:40.482311 19396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:40.482736 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19328
I20260927 18:36:40.482815 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:36:40.482941 19396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993fe230d2264da3834e747131523981" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36611 } }
I20260927 18:36:40.483016 5727 external_mini_cluster.cc:1428] Setting key 2fd302f3e524a571a4e70518ee0db659
I20260927 18:36:40.483048 19396 leader_election.cc:302] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993fe230d2264da3834e747131523981; no voters:
I20260927 18:36:40.483210 19396 leader_election.cc:288] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:36:40.483265 19399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:40.483410 19399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [term 1 LEADER]: Becoming Leader. State: Replica: 993fe230d2264da3834e747131523981, State: Running, Role: LEADER
I20260927 18:36:40.483474 19396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:40.483500 19399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993fe230d2264da3834e747131523981" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36611 } }
I20260927 18:36:40.483831 19401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "993fe230d2264da3834e747131523981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993fe230d2264da3834e747131523981" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36611 } } }
I20260927 18:36:40.483923 19401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:40.483963 19400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993fe230d2264da3834e747131523981. Latest consensus state: current_term: 1 leader_uuid: "993fe230d2264da3834e747131523981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993fe230d2264da3834e747131523981" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 36611 } } }
I20260927 18:36:40.484057 19400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981 [sys.catalog]: This master's current role is: LEADER
W20260927 18:36:40.485390 19416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:36:40.485446 19416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:36:40.485535 19407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:40.485742 19407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:40.487202 19407 catalog_manager.cc:1347] Generated new cluster ID: 2b7ffee80f0243bbae6740900ebc103d
I20260927 18:36:40.487253 19407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:36:40 dist-test-slave-19nl krb5kdc[19315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:40.495913 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:40.483574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.5.151.254:36611)
0927 18:36:40.484281 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:40.484288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:40.484312 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:36:40.484440 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:40.484459 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:40.484516 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:40.484673 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:40.485459 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:40.486896 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:40.487984 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:40.488635 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:40.488760 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:40.491588 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:40.491592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:40.491592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:40.491604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:40.494133 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:40.494570 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:40.494573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:40.494573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:40.494691 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:40.494909 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:40.494910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:40.494910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:40.495200 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:40.495349 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:40.495465 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 18:36:40.495591 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":64,"threads_started":1}
I20260927 18:36:40.496873 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:36:40.500171 19407 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:36:40.500687 19407 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:40.503388 19407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 993fe230d2264da3834e747131523981: Generated new TSK 0
I20260927 18:36:40.503561 19407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:36:40.550884 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19328
2026-09-27T18:36:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1288 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:36:40 dist-test-slave-19nl krb5kdc[19423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:36:40 dist-test-slave-19nl krb5kdc[19423](info): set up 2 sockets
Sep 27 18:36:40 dist-test-slave-19nl krb5kdc[19423](info): commencing operation
krb5kdc: starting...
W20260927 18:36:41.554636 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:36:41 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:36:41.667683 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32911
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:41.747643 19436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:41.747830 19436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:41.747851 19436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:41.748976 19436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:41.749012 19436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:41.749025 19436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:41.749037 19436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:41.749084 19436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:41.750581 19436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:41.750816 19436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:41.751044 19436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:41.755695 19442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:41.755724 19436 server_base.cc:1034] running on GCE node
W20260927 18:36:41.755703 19444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:41.755703 19441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:41.756129 19436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:41.756407 19436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:41.757570 19436 hybrid_clock.cc:648] HybridClock initialized: now 1790534201757558 us; error 43 us; skew 500 ppm
Sep 27 18:36:41 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:41.759821 19436 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:41.760300 19436 webserver.cc:460] Webserver started at http://127.5.151.254:41647/ using document root <none> and password file <none>
I20260927 18:36:41.760553 19436 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:41.760618 19436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:41.760748 19436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:41.761714 19436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "129b6a8a23af4e599e3e0077c2eba34a"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "77c1f2310ec900c27b653c58d5b81159"
server_key_iv: "07e62b93a6054da6e8796c956c847b56"
server_key_version: "encryptionkey@0"
I20260927 18:36:41.762072 19436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "129b6a8a23af4e599e3e0077c2eba34a"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "77c1f2310ec900c27b653c58d5b81159"
server_key_iv: "07e62b93a6054da6e8796c956c847b56"
server_key_version: "encryptionkey@0"
I20260927 18:36:41.763386 19436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 18:36:41.764034 19451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:41.764221 19436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:41.764313 19436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "129b6a8a23af4e599e3e0077c2eba34a"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "77c1f2310ec900c27b653c58d5b81159"
server_key_iv: "07e62b93a6054da6e8796c956c847b56"
server_key_version: "encryptionkey@0"
I20260927 18:36:41.764394 19436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:41.779508 19436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:41.781315 19436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:41.781492 19436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:41.785570 19436 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:32911
I20260927 18:36:41.785616 19503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:32911 every 8 connection(s)
I20260927 18:36:41.785995 19436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:41.787055 19504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:41.789245 19504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Bootstrap starting.
I20260927 18:36:41.789763 19504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:41.789996 19504 log.cc:824] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:41.790727 19504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: No bootstrap required, opened a new log
I20260927 18:36:41.792233 19504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:41.792426 19504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:41.792467 19504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129b6a8a23af4e599e3e0077c2eba34a, State: Initialized, Role: FOLLOWER
I20260927 18:36:41.792481 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19436
I20260927 18:36:41.792573 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:36:41.792618 19504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:41.792704 19504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:41.792737 19504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:41.792785 5727 external_mini_cluster.cc:1428] Setting key 5debd81b24e32ae8514f1672ff923b73
I20260927 18:36:41.792783 19504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:41.793761 19504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:41.793875 19504 leader_election.cc:302] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129b6a8a23af4e599e3e0077c2eba34a; no voters:
I20260927 18:36:41.794078 19504 leader_election.cc:288] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:36:41.794128 19509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:41.794281 19509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 LEADER]: Becoming Leader. State: Replica: 129b6a8a23af4e599e3e0077c2eba34a, State: Running, Role: LEADER
I20260927 18:36:41.794349 19504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:41.794380 19509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:41.794677 19509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129b6a8a23af4e599e3e0077c2eba34a. Latest consensus state: current_term: 1 leader_uuid: "129b6a8a23af4e599e3e0077c2eba34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } } }
I20260927 18:36:41.794776 19509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:41.794687 19511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "129b6a8a23af4e599e3e0077c2eba34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } } }
I20260927 18:36:41.794919 19511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:41.795123 19515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:41.795852 19515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:41.797847 19515 catalog_manager.cc:1347] Generated new cluster ID: e85e822c58cd440fa267bb6e1332e463
I20260927 18:36:41.797935 19515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:36:41 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:41.803858 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:41.793243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.5.151.254:32911)
0927 18:36:41.793414 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:41.793422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:41.793454 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:36:41.793542 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:41.793547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:41.793585 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:41.793688 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:41.794579 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:41.796092 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:41.797137 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:41.797604 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:41.797729 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:41.800360 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:41.800363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:41.800364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:41.800374 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:41.802222 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:41.802661 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:41.802663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:41.802664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:41.802768 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:41.802982 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:41.802983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:41.802983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:41.803254 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:41.803395 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:41.803524 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:36:41.803642 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260927 18:36:41.807552 19515 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:36:41.808130 19515 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:41.822453 19515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Generated new TSK 0
I20260927 18:36:41.822661 19515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:36:41.848927 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:32911
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:41.928129 19530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:41.928329 19530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:41.928354 19530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:41.929518 19530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:41.929563 19530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:41.929589 19530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:36:41.931185 19530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:41.931442 19530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:41.931699 19530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:41.936657 19536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:41.936725 19530 server_base.cc:1034] running on GCE node
W20260927 18:36:41.936657 19535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:41.936685 19538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:41.937057 19530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:41.937256 19530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:41.938395 19530 hybrid_clock.cc:648] HybridClock initialized: now 1790534201938383 us; error 27 us; skew 500 ppm
Sep 27 18:36:41 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:41.940810 19530 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:36:41.941267 19530 webserver.cc:460] Webserver started at http://127.5.151.193:39311/ using document root <none> and password file <none>
I20260927 18:36:41.941520 19530 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:41.941565 19530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:41.941721 19530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:41.942827 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "5788c4cecce6416aad694994dec6ea79"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "086520a23b4c11a55a5b11cb4428d983"
server_key_iv: "b157f56643633d7689777250aa2ae555"
server_key_version: "encryptionkey@0"
I20260927 18:36:41.943179 19530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "5788c4cecce6416aad694994dec6ea79"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "086520a23b4c11a55a5b11cb4428d983"
server_key_iv: "b157f56643633d7689777250aa2ae555"
server_key_version: "encryptionkey@0"
I20260927 18:36:41.944566 19530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:36:41.945271 19545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:41.945444 19530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:41.945547 19530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "5788c4cecce6416aad694994dec6ea79"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "086520a23b4c11a55a5b11cb4428d983"
server_key_iv: "b157f56643633d7689777250aa2ae555"
server_key_version: "encryptionkey@0"
I20260927 18:36:41.945619 19530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:41.956228 19530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:41.957898 19530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:41.958046 19530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:41.958246 19530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:41.958616 19530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:41.958674 19530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:41.958712 19530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:41.958739 19530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:41.964406 19530 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:33997
I20260927 18:36:41.964447 19658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:33997 every 8 connection(s)
I20260927 18:36:41.964736 19530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:36:41 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534201, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:41.973470 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:41.965424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:49193 (local address 127.5.151.254:32911)
0927 18:36:41.965466 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:41.965468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:41.965673 (+ 205us) server_negotiation.cc:407] Connection header received
0927 18:36:41.966119 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:41.966120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:41.966132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:41.966185 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:41.967131 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:41.967667 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:41.968620 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:41.968853 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:41.968911 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:41.971105 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:41.971106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:41.971107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:41.971113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:41.972329 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:41.972688 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:41.972689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:41.972689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:41.972738 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:41.972943 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:41.972944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:41.972944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:41.973043 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:41.973092 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:41.973320 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 18:36:41.973383 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:36:41.973608 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:41.965384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:49193)
0927 18:36:41.965554 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:41.965593 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:41.966039 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:41.966236 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:41.966239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:41.966847 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:41.967023 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:41.967027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:41.967757 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:41.967757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:41.968544 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:41.968546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:41.968937 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:41.968938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:41.969036 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:41.969573 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:41.969581 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:41.971018 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:41.972401 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:41.972408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:41.972410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:41.972624 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:41.972794 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:41.972794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:41.972794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:41.972888 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:41.973097 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:41.973101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:41.973212 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 18:36:41.973292 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:36:41.973407 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260927 18:36:41.973973 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19530
I20260927 18:36:41.974097 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:36:41.974195 19659 heartbeater.cc:347] Connected to a master server at 127.5.151.254:32911
I20260927 18:36:41.974303 19659 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:41.974360 5727 external_mini_cluster.cc:1428] Setting key 224f0a8811663b8f70713be16e02f3a9
I20260927 18:36:41.974522 19659 heartbeater.cc:511] Master 127.5.151.254:32911 requested a full tablet report, sending...
I20260927 18:36:41.975147 19467 ts_manager.cc:194] Registered new tserver with Master: 5788c4cecce6416aad694994dec6ea79 (127.5.151.193:33997)
I20260927 18:36:41.975765 19467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:49193
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:36:41.999262 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:32911
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:42.078840 19664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:42.079023 19664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:42.079044 19664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:42.080054 19664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:42.080085 19664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:42.080111 19664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:36:42.081382 19664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:42.081585 19664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:42.081732 19664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:42.086352 19672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.086370 19669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.086361 19670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:42.086712 19664 server_base.cc:1034] running on GCE node
I20260927 18:36:42.086930 19664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:42.087112 19664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:42.088243 19664 hybrid_clock.cc:648] HybridClock initialized: now 1790534202088235 us; error 30 us; skew 500 ppm
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:42.090462 19664 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:36:42.090847 19664 webserver.cc:460] Webserver started at http://127.5.151.194:45367/ using document root <none> and password file <none>
I20260927 18:36:42.091064 19664 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:42.091107 19664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:42.091238 19664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:42.092173 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "d4d84a8c1eb74614bc512ecd33b5aedb"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "23fc55c3dc6814b9ecc198ca282969e2"
server_key_iv: "59eb45632bb70a2d44355417f14543e0"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.092502 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "d4d84a8c1eb74614bc512ecd33b5aedb"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "23fc55c3dc6814b9ecc198ca282969e2"
server_key_iv: "59eb45632bb70a2d44355417f14543e0"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.093730 19664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:36:42.094409 19679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.094619 19664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:36:42.094703 19664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "d4d84a8c1eb74614bc512ecd33b5aedb"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "23fc55c3dc6814b9ecc198ca282969e2"
server_key_iv: "59eb45632bb70a2d44355417f14543e0"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.094769 19664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:42.115427 19664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:42.117110 19664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:42.117265 19664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:42.117487 19664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:42.117780 19664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:42.117832 19664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.117873 19664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:42.117909 19664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.123245 19664 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:35031
I20260927 18:36:42.123332 19792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:35031 every 8 connection(s)
I20260927 18:36:42.123610 19664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:42.131785 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.124323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:60927 (local address 127.5.151.254:32911)
0927 18:36:42.124389 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.124392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.124577 (+ 185us) server_negotiation.cc:407] Connection header received
0927 18:36:42.125034 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.125035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.125045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.125109 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:42.125976 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.126436 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.127394 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.127572 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.127623 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.129575 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:42.129576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:42.129576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:42.129581 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:42.130776 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.131117 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.131118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.131118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.131164 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.131346 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.131346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.131346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.131433 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:42.131477 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.131672 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.131730 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:36:42.131922 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.124300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:60927)
0927 18:36:42.124464 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.124498 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.124929 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.125159 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.125162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.125717 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:42.125905 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.125909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.126513 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.126514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.127323 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.127326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.127642 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.127643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.127734 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.128219 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:42.128227 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:42.129501 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:42.130840 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.130846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.130847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.131060 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.131215 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.131215 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.131216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.131300 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.131481 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:42.131484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:42.131578 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.131649 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.131750 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260927 18:36:42.132462 19793 heartbeater.cc:347] Connected to a master server at 127.5.151.254:32911
I20260927 18:36:42.132562 19793 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:42.132722 19793 heartbeater.cc:511] Master 127.5.151.254:32911 requested a full tablet report, sending...
I20260927 18:36:42.133126 19467 ts_manager.cc:194] Registered new tserver with Master: d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031)
I20260927 18:36:42.133492 19467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:60927
I20260927 18:36:42.133685 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19664
I20260927 18:36:42.133787 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:36:42.133991 5727 external_mini_cluster.cc:1428] Setting key 09d67fe9f6423e93c6ebb2e0020343c8
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:36:42.157718 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:32911
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:42.236657 19798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:42.236865 19798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:42.236888 19798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:42.238021 19798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:42.238062 19798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:42.238113 19798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:36:42.239486 19798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:42.239723 19798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:42.239933 19798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:42.244719 19804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.244699 19803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.244697 19806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:42.244881 19798 server_base.cc:1034] running on GCE node
I20260927 18:36:42.245123 19798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:42.245316 19798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:42.246459 19798 hybrid_clock.cc:648] HybridClock initialized: now 1790534202246449 us; error 25 us; skew 500 ppm
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:42.249030 19798 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:36:42.249431 19798 webserver.cc:460] Webserver started at http://127.5.151.195:34779/ using document root <none> and password file <none>
I20260927 18:36:42.249640 19798 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:42.249703 19798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:42.249845 19798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:42.250885 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "10847f067c4a42448493cdbdadb85eb7"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "1a0845f57f7bec4b9887d2ff29068018"
server_key_iv: "0ae59c150dd07ba964fbfefdfba0fbbf"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.251221 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "10847f067c4a42448493cdbdadb85eb7"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "1a0845f57f7bec4b9887d2ff29068018"
server_key_iv: "0ae59c150dd07ba964fbfefdfba0fbbf"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.252519 19798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:36:42.253151 19813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.253304 19798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:42.253399 19798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "10847f067c4a42448493cdbdadb85eb7"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "1a0845f57f7bec4b9887d2ff29068018"
server_key_iv: "0ae59c150dd07ba964fbfefdfba0fbbf"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.253476 19798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:42.272923 19798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:42.275189 19798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:42.275345 19798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:42.275584 19798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:42.275888 19798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:42.275938 19798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.275982 19798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:42.276021 19798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.281386 19798 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:44701
I20260927 18:36:42.281481 19926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:44701 every 8 connection(s)
I20260927 18:36:42.281775 19798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:42.282164 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19798
I20260927 18:36:42.282279 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:36:42.282495 5727 external_mini_cluster.cc:1428] Setting key 30226fdf5551c661b2adf8d5032caa32
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:42.290649 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.282743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:42645 (local address 127.5.151.254:32911)
0927 18:36:42.282790 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.282792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.282877 (+ 85us) server_negotiation.cc:407] Connection header received
0927 18:36:42.283562 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.283563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.283572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.283609 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:42.284522 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.285000 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.285955 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.286128 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.286176 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.288147 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:42.288148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:42.288148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:42.288154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:42.289507 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.289829 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.289830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.289830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.289876 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.290075 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.290076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.290076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.290171 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:42.290218 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.290461 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.290539 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:36:42.290771 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.282501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:42645)
0927 18:36:42.282720 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.282766 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.283442 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.283663 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.283666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.284241 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:42.284451 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.284456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.285073 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.285073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.285885 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.285887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.286199 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.286200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.286297 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.286780 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:42.286789 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:42.288067 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:42.289572 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.289576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.289578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.289768 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.289935 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.289935 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.289936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.290025 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.290222 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:42.290226 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:42.290328 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.290405 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.290514 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":124,"threads_started":1}
I20260927 18:36:42.291231 19927 heartbeater.cc:347] Connected to a master server at 127.5.151.254:32911
I20260927 18:36:42.291309 19927 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:42.291498 19927 heartbeater.cc:511] Master 127.5.151.254:32911 requested a full tablet report, sending...
I20260927 18:36:42.291859 19467 ts_manager.cc:194] Registered new tserver with Master: 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
I20260927 18:36:42.292219 19467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:42645
I20260927 18:36:42.294152 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:36:42.300949 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.295532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.5.151.254:32911)
0927 18:36:42.295610 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.295612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.295817 (+ 205us) server_negotiation.cc:407] Connection header received
0927 18:36:42.295853 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.295853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.295861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.295913 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:42.296420 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.296851 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.297619 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.297784 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.297835 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.299001 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:42.299002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:42.299003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:42.299008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:42.300066 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.300342 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.300343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.300343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.300417 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.300561 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.300562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.300562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.300669 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:42.300735 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.300814 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.300864 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:36:42.302526 19467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:36:42.304101 19467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:36:42.310328 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.306660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.5.151.195:44701)
0927 18:36:42.306820 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.306825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.306834 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:36:42.306893 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.306896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.306910 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.306985 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:42.307133 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.308216 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.309419 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.309891 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.309993 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.310048 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.310214 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.310251 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260927 18:36:42.310318 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.306325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:44701 (local address 127.0.0.1:38160)
0927 18:36:42.306435 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.306445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.306494 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.306995 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.306996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.306998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:42.307063 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.307065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.308316 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.308317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.309344 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.309346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.309997 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.309998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.310077 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.310103 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.310198 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.310240 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":40,"thread_start_us":48,"threads_started":1}
I20260927 18:36:42.310353 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.306227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.5.151.194:35031)
0927 18:36:42.306396 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.306402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.306410 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:36:42.306445 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.306447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.306463 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.306537 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:42.306793 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.307837 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.309176 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.309893 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.309974 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.310034 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.310235 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.310276 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260927 18:36:42.310324 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.306464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:33997 (local address 127.0.0.1:40884)
0927 18:36:42.306674 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.306684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.306732 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.307018 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.307019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.307021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:42.307073 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.307075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.308359 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.308360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.309243 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.309245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.310077 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.310078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.310142 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.310147 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.310176 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.310246 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260927 18:36:42.310602 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.306124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:35031 (local address 127.0.0.1:50154)
0927 18:36:42.306226 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.306253 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.306325 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.306650 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.306653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.306656 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:42.306717 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.306718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.307941 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.307942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.309108 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.309110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.310067 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.310069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.310148 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.310151 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.310194 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.310479 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":231,"thread_start_us":65,"threads_started":1}
I20260927 18:36:42.310614 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.306601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.5.151.193:33997)
0927 18:36:42.306849 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.306853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.306861 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:36:42.306895 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.306907 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.306921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.307171 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:42.307235 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.308286 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.309326 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.309991 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.310363 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.310437 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.310462 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.310501 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260927 18:36:42.311229 19593 tablet_service.cc:1467] Processing CreateTablet for tablet 1cce49c3cddf4e539621ba8a975b9459 (DEFAULT_TABLE table=test-table [id=965d3b8cf40041379bbb8452fc431148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:42.311576 19593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cce49c3cddf4e539621ba8a975b9459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:42.311528 19861 tablet_service.cc:1467] Processing CreateTablet for tablet 1cce49c3cddf4e539621ba8a975b9459 (DEFAULT_TABLE table=test-table [id=965d3b8cf40041379bbb8452fc431148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:42.311822 19861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cce49c3cddf4e539621ba8a975b9459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:42.313889 19941 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Bootstrap starting.
I20260927 18:36:42.314021 19942 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Bootstrap starting.
I20260927 18:36:42.314463 19727 tablet_service.cc:1467] Processing CreateTablet for tablet 1cce49c3cddf4e539621ba8a975b9459 (DEFAULT_TABLE table=test-table [id=965d3b8cf40041379bbb8452fc431148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:42.314694 19941 tablet_bootstrap.cc:654] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:42.314792 19727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cce49c3cddf4e539621ba8a975b9459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:42.314844 19942 tablet_bootstrap.cc:654] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:42.315235 19942 log.cc:824] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:42.315245 19941 log.cc:824] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:42.315872 19942 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: No bootstrap required, opened a new log
I20260927 18:36:42.315964 19942 ts_tablet_manager.cc:1397] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:36:42.317248 19945 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Bootstrap starting.
I20260927 18:36:42.317342 19942 raft_consensus.cc:348] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.317524 19942 raft_consensus.cc:374] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:42.317567 19942 raft_consensus.cc:729] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10847f067c4a42448493cdbdadb85eb7, State: Initialized, Role: FOLLOWER
I20260927 18:36:42.317678 19942 consensus_queue.cc:260] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.317773 19941 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: No bootstrap required, opened a new log
I20260927 18:36:42.317850 19941 ts_tablet_manager.cc:1397] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 18:36:42.317951 19942 ts_tablet_manager.cc:1428] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:36:42.318071 19927 heartbeater.cc:503] Master 127.5.151.254:32911 was elected leader, sending a full tablet report...
I20260927 18:36:42.318252 19945 tablet_bootstrap.cc:654] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:42.318827 19945 log.cc:824] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:42.319677 19945 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: No bootstrap required, opened a new log
I20260927 18:36:42.319726 19945 ts_tablet_manager.cc:1397] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:36:42.319768 19941 raft_consensus.cc:348] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.319921 19941 raft_consensus.cc:374] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:42.319983 19941 raft_consensus.cc:729] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5788c4cecce6416aad694994dec6ea79, State: Initialized, Role: FOLLOWER
I20260927 18:36:42.320084 19941 consensus_queue.cc:260] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.320325 19941 ts_tablet_manager.cc:1428] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 18:36:42.320379 19659 heartbeater.cc:503] Master 127.5.151.254:32911 was elected leader, sending a full tablet report...
I20260927 18:36:42.321579 19945 raft_consensus.cc:348] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.321724 19945 raft_consensus.cc:374] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:42.321769 19945 raft_consensus.cc:729] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d84a8c1eb74614bc512ecd33b5aedb, State: Initialized, Role: FOLLOWER
I20260927 18:36:42.321883 19945 consensus_queue.cc:260] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.322175 19945 ts_tablet_manager.cc:1428] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:36:42.322208 19793 heartbeater.cc:503] Master 127.5.151.254:32911 was elected leader, sending a full tablet report...
W20260927 18:36:42.374146 19794 tablet.cc:2369] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:36:42.433666 19948 raft_consensus.cc:482] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:42.433791 19948 raft_consensus.cc:504] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.434226 19948 leader_election.cc:288] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
I20260927 18:36:42.438062 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.434660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:43357 (local address 127.5.151.195:44701)
0927 18:36:42.434710 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.434712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.434963 (+ 251us) server_negotiation.cc:407] Connection header received
0927 18:36:42.435019 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.435020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.435029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.435081 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:42.435234 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.435693 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.437213 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.437604 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.437783 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.437796 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.437955 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.437997 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:36:42.438078 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.434706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:44701 (local address 127.5.151.193:43357)
0927 18:36:42.434899 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.434910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.434962 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.435096 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.435097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.435098 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:42.435167 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.435168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.435778 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.435779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.437089 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.437092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.437782 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.437783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.437872 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.437876 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.437948 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.437991 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":10,"thread_start_us":119,"threads_started":1}
I20260927 18:36:42.438378 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.434574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:44099 (local address 127.5.151.194:35031)
0927 18:36:42.434619 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.434621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.434651 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:36:42.434679 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.434679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.434689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.434736 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:42.434977 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.435831 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.437193 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.437910 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.437997 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.438006 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.438257 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.438303 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:36:42.438392 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.434441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:35031 (local address 127.5.151.193:44099)
0927 18:36:42.434502 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.434517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.434611 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.434752 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.434753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.434754 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:42.434863 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.434866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.435923 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.435924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.437088 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.437092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.438047 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.438049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.438166 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.438171 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.438237 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.438291 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":49}
I20260927 18:36:42.438771 19747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cce49c3cddf4e539621ba8a975b9459" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" is_pre_election: true
I20260927 18:36:42.438796 19881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cce49c3cddf4e539621ba8a975b9459" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10847f067c4a42448493cdbdadb85eb7" is_pre_election: true
I20260927 18:36:42.438936 19747 raft_consensus.cc:2454] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 0.
I20260927 18:36:42.438956 19881 raft_consensus.cc:2454] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 0.
I20260927 18:36:42.439167 19549 leader_election.cc:302] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79, d4d84a8c1eb74614bc512ecd33b5aedb; no voters:
I20260927 18:36:42.439296 19948 raft_consensus.cc:2781] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:36:42.439358 19948 raft_consensus.cc:482] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:36:42.439393 19948 raft_consensus.cc:3037] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:42.440085 19948 raft_consensus.cc:504] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.440213 19948 leader_election.cc:288] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 election: Requested vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
I20260927 18:36:42.440402 19747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cce49c3cddf4e539621ba8a975b9459" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb"
I20260927 18:36:42.440477 19747 raft_consensus.cc:3037] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:42.441181 19747 raft_consensus.cc:2454] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 1.
I20260927 18:36:42.441357 19549 leader_election.cc:302] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79, d4d84a8c1eb74614bc512ecd33b5aedb; no voters:
I20260927 18:36:42.441377 19881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cce49c3cddf4e539621ba8a975b9459" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10847f067c4a42448493cdbdadb85eb7"
I20260927 18:36:42.441485 19948 raft_consensus.cc:2781] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:42.441443 19881 raft_consensus.cc:3037] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:42.441670 19948 raft_consensus.cc:686] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 1 LEADER]: Becoming Leader. State: Replica: 5788c4cecce6416aad694994dec6ea79, State: Running, Role: LEADER
I20260927 18:36:42.441779 19948 consensus_queue.cc:237] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.442435 19881 raft_consensus.cc:2454] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 1.
I20260927 18:36:42.442785 19467 catalog_manager.cc:5495] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5788c4cecce6416aad694994dec6ea79 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "5788c4cecce6416aad694994dec6ea79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } health_report { overall_health: HEALTHY } } }
I20260927 18:36:42.455137 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.452191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.5.151.193:33997)
0927 18:36:42.452226 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.452228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.452239 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:36:42.452312 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.452313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.452332 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.452389 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:36:42.452544 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.452994 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.454296 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.454489 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.454595 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.454763 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:36:42.454857 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:36:42.454920 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.455023 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.455040 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:36:42.457152 19881 raft_consensus.cc:1261] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Refusing update from remote peer 5788c4cecce6416aad694994dec6ea79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:42.457454 19747 raft_consensus.cc:1261] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Refusing update from remote peer 5788c4cecce6416aad694994dec6ea79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:42.457481 19951 consensus_queue.cc:1035] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:42.457700 19951 consensus_queue.cc:1035] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:42.461970 19467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:42.462071 19467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:42.462682 19467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Sending DeleteTablet for 3 replicas of tablet 1cce49c3cddf4e539621ba8a975b9459
I20260927 18:36:42.462980 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19530
I20260927 18:36:42.463024 19593 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cce49c3cddf4e539621ba8a975b9459 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:42 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:40884
I20260927 18:36:42.463574 19727 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cce49c3cddf4e539621ba8a975b9459 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:42 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50154
I20260927 18:36:42.463627 19861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cce49c3cddf4e539621ba8a975b9459 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:42 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:38160
I20260927 18:36:42.466830 19962 tablet_replica.cc:330] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: stopping tablet replica
I20260927 18:36:42.467013 19963 tablet_replica.cc:330] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: stopping tablet replica
I20260927 18:36:42.467159 19963 raft_consensus.cc:2229] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:42.467273 19963 raft_consensus.cc:2258] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:42.467429 19962 raft_consensus.cc:2229] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:42.467501 19962 pending_rounds.cc:62] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Trying to abort 2 pending ops.
I20260927 18:36:42.467555 19962 pending_rounds.cc:68] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334028093261733888 op_type: WRITE_OP write_request { tablet_id: "1cce49c3cddf4e539621ba8a975b9459" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\346\306\345\325\006\"8\n\ntest-admin\022*\n 965d3b8cf40041379bbb8452fc431148\020\001\030\001 \001(\001" signature: "<redacted>""\242\275)\205\030\255(ec:\3761\350\'\275>\000\244\305\301\330\277B\026\243\004r\215\373e\270\364!\331\004\302b\250\030$H\331[OC\200m\336\355\rx\370Zv{O\352\330l|T\377\034\360" signing_key_seq_num: 0 } } request_id { client_id: "5587bfdae6344e23a0c00fd540f5da95" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 18:36:42.467903 19962 pending_rounds.cc:68] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334028093202391040 op_type: NO_OP noop_request { }
I20260927 18:36:42.467979 19962 raft_consensus.cc:2866] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 18:36:42.468056 19962 raft_consensus.cc:2258] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:42.468328 19963 ts_tablet_manager.cc:1905] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:42.468509 19962 ts_tablet_manager.cc:1905] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:36:42.468888 19681 connection.cc:535] server connection from 127.5.151.193:44099 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 18:36:42.469282 19455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:33997 (error 108)
I20260927 18:36:42.470017 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.469905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:33997 (local address 127.0.0.1:40910)
0927 18:36:42.469951 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.469982 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 18:36:42.470242 19455 catalog_manager.cc:4584] TS 5788c4cecce6416aad694994dec6ea79 (127.5.151.193:33997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cce49c3cddf4e539621ba8a975b9459: Network error: Client connection negotiation failed: client connection to 127.5.151.193:33997: connect: Connection refused (error 111)
I20260927 18:36:42.470383 19962 ts_tablet_manager.cc:1918] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:42.470438 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19664
I20260927 18:36:42.470463 19962 log.cc:1196] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/1cce49c3cddf4e539621ba8a975b9459
I20260927 18:36:42.470983 19963 ts_tablet_manager.cc:1918] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:42.471101 19963 log.cc:1196] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/1cce49c3cddf4e539621ba8a975b9459
I20260927 18:36:42.471413 19963 ts_tablet_manager.cc:1939] T 1cce49c3cddf4e539621ba8a975b9459 P 10847f067c4a42448493cdbdadb85eb7: Deleting consensus metadata
I20260927 18:36:42.471797 19455 catalog_manager.cc:4841] TS 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701): tablet 1cce49c3cddf4e539621ba8a975b9459 (table test-table [id=965d3b8cf40041379bbb8452fc431148]) successfully deleted
I20260927 18:36:42.475282 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19798
I20260927 18:36:42.476809 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.475355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:35031 (local address 127.0.0.1:50168)
0927 18:36:42.476740 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.476772 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.194:35031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1364}
I20260927 18:36:42.479640 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19436
I20260927 18:36:42.484596 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32911
--webserver_interface=127.5.151.254
--webserver_port=41647
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:42.564455 19965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:42.564656 19965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:42.564677 19965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:42.565754 19965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:42.565789 19965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:42.565804 19965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:42.565816 19965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:42.565828 19965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:42.567212 19965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=41647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:42.567394 19965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:42.567575 19965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:42.575950 19972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.575928 19974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:42.576139 19965 server_base.cc:1034] running on GCE node
W20260927 18:36:42.575929 19971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:42.576467 19965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:42.576749 19965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:42.577908 19965 hybrid_clock.cc:648] HybridClock initialized: now 1790534202577892 us; error 46 us; skew 500 ppm
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:42.580361 19965 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:42.580775 19965 webserver.cc:460] Webserver started at http://127.5.151.254:41647/ using document root <none> and password file <none>
I20260927 18:36:42.581014 19965 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:42.581094 19965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:42.582414 19965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:42.583119 19981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.583288 19965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.583389 19965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "129b6a8a23af4e599e3e0077c2eba34a"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "77c1f2310ec900c27b653c58d5b81159"
server_key_iv: "07e62b93a6054da6e8796c956c847b56"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.583781 19965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:42.612905 19965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:42.614758 19965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:42.614945 19965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:42.618728 19965 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:32911
I20260927 18:36:42.618777 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:32911 every 8 connection(s)
I20260927 18:36:42.619230 19965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:42.620828 20034 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:36:42.621487 20034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Bootstrap starting.
I20260927 18:36:42.629349 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 19965
I20260927 18:36:42.629367 20034 log.cc:824] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:42.629658 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:33997
--local_ip_for_outbound_sockets=127.5.151.193
--tserver_master_addrs=127.5.151.254:32911
--webserver_port=39311
--webserver_interface=127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:42.631246 20034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:36:42.631536 20034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Bootstrap complete.
I20260927 18:36:42.633739 20034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:42.633987 20034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129b6a8a23af4e599e3e0077c2eba34a, State: Initialized, Role: FOLLOWER
I20260927 18:36:42.634158 20034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:42.634230 20034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:42.634285 20034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:42.634348 20034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:36:42.635473 20034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:42.635587 20034 leader_election.cc:302] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129b6a8a23af4e599e3e0077c2eba34a; no voters:
I20260927 18:36:42.635787 20034 leader_election.cc:288] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:36:42.635821 20038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:36:42.635933 20038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 LEADER]: Becoming Leader. State: Replica: 129b6a8a23af4e599e3e0077c2eba34a, State: Running, Role: LEADER
I20260927 18:36:42.636025 20038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:42.636070 20034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:42.636270 20038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129b6a8a23af4e599e3e0077c2eba34a. Latest consensus state: current_term: 2 leader_uuid: "129b6a8a23af4e599e3e0077c2eba34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } } }
I20260927 18:36:42.636343 20038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:42.636276 20039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "129b6a8a23af4e599e3e0077c2eba34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } } }
I20260927 18:36:42.636396 20039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:42.636689 20042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:42.637511 20042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:42.637629 20042 catalog_manager.cc:1259] Loaded cluster ID: e85e822c58cd440fa267bb6e1332e463
I20260927 18:36:42.637893 20042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:36:42.638121 20042 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 18:36:42.640866 20042 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 18:36:42.647480 20042 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:36:42.647672 20042 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Loaded TSK: 0
I20260927 18:36:42.647882 20042 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:36:42.733021 20037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:42.733254 20037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:42.733295 20037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:42.734439 20037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:42.734505 20037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:42.734550 20037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:36:42.736063 20037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:33997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=39311
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:42.736305 20037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:42.736565 20037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:36:42.741657 20037 server_base.cc:1034] running on GCE node
W20260927 18:36:42.741783 20061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.741665 20062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.741832 20064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:42.742131 20037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:42.742390 20037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:42.743547 20037 hybrid_clock.cc:648] HybridClock initialized: now 1790534202743518 us; error 46 us; skew 500 ppm
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:42.746016 20037 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:36:42.746522 20037 webserver.cc:460] Webserver started at http://127.5.151.193:39311/ using document root <none> and password file <none>
I20260927 18:36:42.746786 20037 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:42.746865 20037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:42.748201 20037 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:42.748914 20071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.749092 20037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:42.749204 20037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "5788c4cecce6416aad694994dec6ea79"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "086520a23b4c11a55a5b11cb4428d983"
server_key_iv: "b157f56643633d7689777250aa2ae555"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.749544 20037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:42.759406 20037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:42.761473 20037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:42.761618 20037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:42.761838 20037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:42.762277 20078 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:36:42.764215 20037 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:36:42.764274 20037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 18:36:42.764319 20037 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:36:42.764930 20037 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:36:42.764981 20037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 18:36:42.765046 20078 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Bootstrap starting.
I20260927 18:36:42.771377 20037 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:33997
I20260927 18:36:42.771528 20185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:33997 every 8 connection(s)
I20260927 18:36:42.771780 20037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:36:42.774749 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 20037
I20260927 18:36:42.775035 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:35031
--local_ip_for_outbound_sockets=127.5.151.194
--tserver_master_addrs=127.5.151.254:32911
--webserver_port=45367
--webserver_interface=127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:42.775791 20078 log.cc:824] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:42.778460 20078 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:36:42.778789 20078 tablet_bootstrap.cc:492] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Bootstrap complete.
I20260927 18:36:42.780018 20078 ts_tablet_manager.cc:1397] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260927 18:36:42.781471 20078 raft_consensus.cc:348] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.781786 20078 raft_consensus.cc:729] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5788c4cecce6416aad694994dec6ea79, State: Initialized, Role: FOLLOWER
I20260927 18:36:42.781940 20078 consensus_queue.cc:260] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:42.782202 20078 ts_tablet_manager.cc:1428] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:42.790405 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.772532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:57985)
0927 18:36:42.772724 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.772766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.773192 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.780431 (+ 7239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.780435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.780846 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:42.781133 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.781140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.782815 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.782816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.783612 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.783614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.784275 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.784276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.784404 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.784977 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:42.784985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:42.786433 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:42.788716 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.788723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.788725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.788964 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.789250 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.789251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.789251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.789377 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.789761 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:42.789767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:42.789911 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.790021 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.790197 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260927 18:36:42.790421 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.772587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:57985 (local address 127.5.151.254:32911)
0927 18:36:42.776091 (+ 3504us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.776101 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.776144 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:36:42.776275 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.776282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.780224 (+ 3942us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.780392 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:42.781285 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.782717 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.783720 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.784173 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.784298 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.786723 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:42.786728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:42.786728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:42.786740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:42.788604 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.789065 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.789068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.789068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.789171 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.789456 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.789457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.789457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.789687 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:42.789781 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.790074 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.790197 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3449,"thread_start_us":102,"threads_started":1}
I20260927 18:36:42.791535 20186 heartbeater.cc:347] Connected to a master server at 127.5.151.254:32911
I20260927 18:36:42.791636 20186 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:42.791831 20186 heartbeater.cc:511] Master 127.5.151.254:32911 requested a full tablet report, sending...
I20260927 18:36:42.792558 19998 ts_manager.cc:194] Registered new tserver with Master: 5788c4cecce6416aad694994dec6ea79 (127.5.151.193:33997)
I20260927 18:36:42.793715 19998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:57985
I20260927 18:36:42.794737 20186 heartbeater.cc:503] Master 127.5.151.254:32911 was elected leader, sending a full tablet report...
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.193@KRBTEST.COM
I20260927 18:36:42.801951 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.793426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.5.151.193:33997)
0927 18:36:42.793595 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.793600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.794203 (+ 603us) server_negotiation.cc:407] Connection header received
0927 18:36:42.794238 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.794240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.794254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.794333 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:42.794913 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.795910 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.796659 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.796961 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.797041 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.799123 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:42.799131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:42.799131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:42.799138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:42.800679 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.801091 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.801095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.801095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.801151 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.801361 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.801362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.801362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.801498 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:42.801585 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.801760 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.801863 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260927 18:36:42.801950 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.793149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:33997 (local address 127.0.0.1:40916)
0927 18:36:42.794027 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.794062 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.794125 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.794390 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.794395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.794760 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:42.794820 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.794822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.795982 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.795983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.796553 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.796555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.797040 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.797041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.797115 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.797632 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:42.797637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:42.799031 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:42.800782 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.800789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.800791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.801008 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.801215 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.801216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.801216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.801273 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.801567 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:42.801573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:42.801623 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.801724 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.801849 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":114,"threads_started":1}
I20260927 18:36:42.802626 20120 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cce49c3cddf4e539621ba8a975b9459 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:42 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:40916
I20260927 18:36:42.803128 20195 tablet_replica.cc:330] stopping tablet replica
I20260927 18:36:42.803259 20195 raft_consensus.cc:2229] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:42.803339 20195 raft_consensus.cc:2258] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:42.803632 20195 ts_tablet_manager.cc:1905] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:42.805555 20195 ts_tablet_manager.cc:1918] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:42.805644 20195 log.cc:1196] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/1cce49c3cddf4e539621ba8a975b9459
I20260927 18:36:42.805866 20195 ts_tablet_manager.cc:1939] T 1cce49c3cddf4e539621ba8a975b9459 P 5788c4cecce6416aad694994dec6ea79: Deleting consensus metadata
I20260927 18:36:42.806360 19985 catalog_manager.cc:4841] TS 5788c4cecce6416aad694994dec6ea79 (127.5.151.193:33997): tablet 1cce49c3cddf4e539621ba8a975b9459 (table test-table [id=965d3b8cf40041379bbb8452fc431148]) successfully deleted
W20260927 18:36:42.868467 20191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:42.868671 20191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:42.868695 20191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:42.869751 20191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:42.869784 20191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:42.869808 20191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:36:42.871304 20191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:35031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=45367
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:42.871495 20191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:42.871665 20191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:42.876514 20203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.876566 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:42.876514 20201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:42.876719 20191 server_base.cc:1034] running on GCE node
I20260927 18:36:42.876924 20191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:42.877134 20191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:42.878269 20191 hybrid_clock.cc:648] HybridClock initialized: now 1790534202878260 us; error 26 us; skew 500 ppm
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:42.880722 20191 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:36:42.881140 20191 webserver.cc:460] Webserver started at http://127.5.151.194:45367/ using document root <none> and password file <none>
I20260927 18:36:42.881371 20191 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:42.881448 20191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:42.882810 20191 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:42.883424 20210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.883587 20191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:42.883677 20191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "d4d84a8c1eb74614bc512ecd33b5aedb"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "23fc55c3dc6814b9ecc198ca282969e2"
server_key_iv: "59eb45632bb70a2d44355417f14543e0"
server_key_version: "encryptionkey@0"
I20260927 18:36:42.884027 20191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:42.907311 20191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:42.909251 20191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:42.909412 20191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:42.909658 20191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:42.910120 20217 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 18:36:42.911600 20217 ts_tablet_manager.cc:1852] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 18:36:42.911871 20217 ts_tablet_manager.cc:1905] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:42.913648 20217 ts_tablet_manager.cc:1918] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:42.913723 20217 log.cc:1196] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/1cce49c3cddf4e539621ba8a975b9459
I20260927 18:36:42.913937 20217 ts_tablet_manager.cc:1939] T 1cce49c3cddf4e539621ba8a975b9459 P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting consensus metadata
I20260927 18:36:42.914144 20191 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 18:36:42.914211 20191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 18:36:42.914258 20191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:42.914286 20191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:42.920550 20191 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:35031
I20260927 18:36:42.920603 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:35031 every 8 connection(s)
I20260927 18:36:42.920943 20191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:36:42.925882 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 20191
I20260927 18:36:42.926162 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:44701
--local_ip_for_outbound_sockets=127.5.151.195
--tserver_master_addrs=127.5.151.254:32911
--webserver_port=34779
--webserver_interface=127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:36:42 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:42.930330 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.921724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:45137 (local address 127.5.151.254:32911)
0927 18:36:42.921801 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:42.921803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:42.922064 (+ 261us) server_negotiation.cc:407] Connection header received
0927 18:36:42.922543 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:42.922544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:42.922606 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:42.922685 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:42.923280 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.923762 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.924705 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.924902 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.924953 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.927411 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:42.927413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:42.927413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:42.927421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:42.928900 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.929336 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.929337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.929337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.929400 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:42.929660 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:42.929661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:42.929662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:42.929777 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:42.929847 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:42.930140 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 18:36:42.930238 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:36:42.930610 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:42.921710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:45137)
0927 18:36:42.921898 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:42.921940 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:42.922453 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:42.922751 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:42.922755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:42.923013 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:42.923204 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.923208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.923838 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.923838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.924636 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:42.924638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:42.924972 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:42.924972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:42.925068 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:42.925686 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:42.925694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:42.927274 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:42.929023 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.929030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.929032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.929255 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.929469 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:42.929470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:42.929470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:42.929586 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:42.929850 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:42.929857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:42.930007 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 18:36:42.930119 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:36:42.930277 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260927 18:36:42.931142 20325 heartbeater.cc:347] Connected to a master server at 127.5.151.254:32911
I20260927 18:36:42.931255 20325 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:42.931434 20325 heartbeater.cc:511] Master 127.5.151.254:32911 requested a full tablet report, sending...
I20260927 18:36:42.931874 19998 ts_manager.cc:194] Registered new tserver with Master: d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031)
I20260927 18:36:42.932252 19998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:45137
W20260927 18:36:43.007820 20328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:43.008049 20328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:43.008071 20328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:43.009234 20328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:43.009289 20328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:43.009341 20328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:36:43.010881 20328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:44701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=34779
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:43.011111 20328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:43.011348 20328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:43.016180 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.016196 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:43.016268 20328 server_base.cc:1034] running on GCE node
W20260927 18:36:43.016197 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:43.016588 20328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:43.016847 20328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:43.018039 20328 hybrid_clock.cc:648] HybridClock initialized: now 1790534203018033 us; error 35 us; skew 500 ppm
Sep 27 18:36:43 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:43.020545 20328 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:36:43.021056 20328 webserver.cc:460] Webserver started at http://127.5.151.195:34779/ using document root <none> and password file <none>
I20260927 18:36:43.021240 20328 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:43.021291 20328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:43.022583 20328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:43.023146 20343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.023295 20328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.023378 20328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "10847f067c4a42448493cdbdadb85eb7"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "1a0845f57f7bec4b9887d2ff29068018"
server_key_iv: "0ae59c150dd07ba964fbfefdfba0fbbf"
server_key_version: "encryptionkey@0"
I20260927 18:36:43.023682 20328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:43.039216 20328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:43.041327 20328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:43.041436 20328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:43.041596 20328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:43.041903 20328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:43.041941 20328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.041962 20328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:43.042007 20328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.047487 20328 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:44701
I20260927 18:36:43.047557 20456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:44701 every 8 connection(s)
I20260927 18:36:43.047889 20328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:43.051151 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 20328
Sep 27 18:36:43 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:36:43.058070 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.048601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:46999 (local address 127.5.151.254:32911)
0927 18:36:43.048667 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.048668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.048909 (+ 241us) server_negotiation.cc:407] Connection header received
0927 18:36:43.049454 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.049455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.049468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.049526 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:43.050454 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.051019 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.052249 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.052444 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.052529 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.054909 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:43.054910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:43.054910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:43.054917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:43.056629 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:43.056996 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:43.056997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:43.056998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:43.057053 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:43.057340 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:43.057343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:43.057343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:43.057487 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:43.057549 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.057880 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.057959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:36:43.058238 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.051532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.5.151.254:32911)
0927 18:36:43.051634 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.051636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.051658 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:36:43.051724 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.051725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.051733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.051810 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:43.052325 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.052791 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.053890 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.054188 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.054255 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.055554 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:43.055555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:43.055555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:43.055562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:43.057030 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:43.057409 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:43.057410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:43.057410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:43.057455 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:43.057667 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:43.057668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:43.057668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:43.057753 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:43.057928 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.057962 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.058022 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":60,"threads_started":1}
I20260927 18:36:43.058270 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.048587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:46999)
0927 18:36:43.048780 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.048815 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:43.049316 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:43.049601 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:43.049605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:43.050166 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:43.050344 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.050349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.051117 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.051119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.052164 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.052166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.052517 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.052518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.052650 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.053133 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:43.053143 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:43.054808 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:43.056699 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:43.056705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:43.056707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:43.056923 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:43.057119 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:43.057120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:43.057120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:43.057253 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:43.057554 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:43.057560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:43.057730 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 18:36:43.057844 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 18:36:43.058027 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260927 18:36:43.058774 20457 heartbeater.cc:347] Connected to a master server at 127.5.151.254:32911
I20260927 18:36:43.058905 20457 heartbeater.cc:464] Registering TS with master...
I20260927 18:36:43.059089 20457 heartbeater.cc:511] Master 127.5.151.254:32911 requested a full tablet report, sending...
I20260927 18:36:43.059461 19997 ts_manager.cc:194] Registered new tserver with Master: 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
I20260927 18:36:43.059772 19997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:46999
I20260927 18:36:43.070111 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:36:43.073029 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.070493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.5.151.254:32911)
0927 18:36:43.070552 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.070554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.070671 (+ 117us) server_negotiation.cc:407] Connection header received
0927 18:36:43.070799 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.070799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.070847 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.070942 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:36:43.071058 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.071521 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.072281 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.072490 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.072563 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.072724 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:36:43.072821 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:36:43.072917 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.072933 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.072953 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:36:43.073375 19997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:36:43.075001 19997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:36:43.077437 20120 tablet_service.cc:1467] Processing CreateTablet for tablet 9fedf21779c549529ad5ce8c9476073f (DEFAULT_TABLE table=test-table [id=a58981885f94437da856c5778f27807a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:43.077631 20120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fedf21779c549529ad5ce8c9476073f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:43.079023 20078 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Bootstrap starting.
I20260927 18:36:43.079326 20078 tablet_bootstrap.cc:654] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:43.080192 20078 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: No bootstrap required, opened a new log
I20260927 18:36:43.080243 20078 ts_tablet_manager.cc:1397] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:43.080387 20078 raft_consensus.cc:348] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.080464 20078 raft_consensus.cc:374] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:43.080513 20078 raft_consensus.cc:729] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5788c4cecce6416aad694994dec6ea79, State: Initialized, Role: FOLLOWER
I20260927 18:36:43.080552 20078 consensus_queue.cc:260] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.080637 20078 ts_tablet_manager.cc:1428] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.080988 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.076731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:35031 (local address 127.0.0.1:50178)
0927 18:36:43.076780 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.076790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:43.076826 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:43.077401 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:43.077402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:43.077404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:43.077449 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.077450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.078735 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.078736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.079791 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.079793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.080717 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.080718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.080788 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.080806 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:36:43.080857 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 18:36:43.080896 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":22}
I20260927 18:36:43.081075 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.076830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:44701 (local address 127.0.0.1:38162)
0927 18:36:43.077012 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.077022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:43.077072 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:43.077460 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:43.077461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:43.077463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:43.077504 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.077506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.078902 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.078903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.080004 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.080008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.080852 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.080853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.080910 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.080913 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:36:43.080970 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:36:43.081002 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":113,"threads_started":1}
I20260927 18:36:43.081101 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.077136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.5.151.195:44701)
0927 18:36:43.077297 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.077300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.077308 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:36:43.077340 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.077341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.077352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.077463 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:43.077564 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.078808 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.080090 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.080779 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.080853 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.080909 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.080987 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.081027 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260927 18:36:43.081529 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.076856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.5.151.194:35031)
0927 18:36:43.077189 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.077194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.077205 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:36:43.077245 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.077247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.077262 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.077373 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:43.077538 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.078631 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.079909 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.080627 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.080699 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.080757 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.081408 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.081447 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":106,"threads_started":1}
I20260927 18:36:43.081939 20391 tablet_service.cc:1467] Processing CreateTablet for tablet 9fedf21779c549529ad5ce8c9476073f (DEFAULT_TABLE table=test-table [id=a58981885f94437da856c5778f27807a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:43.081984 20259 tablet_service.cc:1467] Processing CreateTablet for tablet 9fedf21779c549529ad5ce8c9476073f (DEFAULT_TABLE table=test-table [id=a58981885f94437da856c5778f27807a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:36:43.082252 20259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fedf21779c549529ad5ce8c9476073f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:43.082253 20391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fedf21779c549529ad5ce8c9476073f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:36:43.084367 20217 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Bootstrap starting.
I20260927 18:36:43.085055 20217 tablet_bootstrap.cc:654] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:43.085345 20217 log.cc:824] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:43.085480 20467 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Bootstrap starting.
I20260927 18:36:43.086114 20217 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: No bootstrap required, opened a new log
I20260927 18:36:43.086153 20467 tablet_bootstrap.cc:654] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Neither blocks nor log segments found. Creating new log.
I20260927 18:36:43.086182 20217 ts_tablet_manager.cc:1397] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:36:43.086687 20467 log.cc:824] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:43.087381 20467 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: No bootstrap required, opened a new log
I20260927 18:36:43.087445 20467 ts_tablet_manager.cc:1397] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:36:43.087720 20217 raft_consensus.cc:348] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.087865 20217 raft_consensus.cc:374] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:43.087906 20217 raft_consensus.cc:729] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d84a8c1eb74614bc512ecd33b5aedb, State: Initialized, Role: FOLLOWER
I20260927 18:36:43.088037 20217 consensus_queue.cc:260] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.088294 20217 ts_tablet_manager.cc:1428] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:36:43.088320 20325 heartbeater.cc:503] Master 127.5.151.254:32911 was elected leader, sending a full tablet report...
I20260927 18:36:43.089468 20467 raft_consensus.cc:348] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.089592 20467 raft_consensus.cc:374] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:36:43.089613 20467 raft_consensus.cc:729] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10847f067c4a42448493cdbdadb85eb7, State: Initialized, Role: FOLLOWER
I20260927 18:36:43.089713 20467 consensus_queue.cc:260] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.089967 20467 ts_tablet_manager.cc:1428] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:36:43.089993 20457 heartbeater.cc:503] Master 127.5.151.254:32911 was elected leader, sending a full tablet report...
W20260927 18:36:43.171496 20326 tablet.cc:2369] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:36:43.179534 20192 raft_consensus.cc:482] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:43.179654 20192 raft_consensus.cc:504] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.180061 20192 leader_election.cc:288] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
I20260927 18:36:43.183009 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.180346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:44701 (local address 127.5.151.193:50647)
0927 18:36:43.180526 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.180536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:43.180608 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:43.180749 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:43.180750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:43.180751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:43.180840 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.180842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.181441 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.181442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.182202 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.182204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.182735 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.182736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.182831 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.182848 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:36:43.182890 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 18:36:43.182920 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":114,"threads_started":1}
I20260927 18:36:43.183017 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.180510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:50647 (local address 127.5.151.195:44701)
0927 18:36:43.180556 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.180558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.180593 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:36:43.180678 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.180679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.180688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.180740 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:43.180915 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.181336 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.182304 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.182660 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.182833 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.182843 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.182895 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.182933 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 18:36:43.183107 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.180239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:35031 (local address 127.5.151.193:47809)
0927 18:36:43.180292 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.180305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:43.180356 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:43.180653 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:43.180654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:43.180656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:36:43.180730 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.180732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.181389 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.181390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.182147 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:43.182149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.182860 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.182861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:43.182943 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.182946 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:36:43.183000 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:36:43.183036 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:36:43.183135 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.180471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:47809 (local address 127.5.151.194:35031)
0927 18:36:43.180523 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.180525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.180536 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:36:43.180566 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.180567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.180577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.180635 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:36:43.180820 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.181313 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.182236 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.182668 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.182751 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.182760 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.183022 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.183062 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:36:43.183398 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fedf21779c549529ad5ce8c9476073f" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10847f067c4a42448493cdbdadb85eb7" is_pre_election: true
I20260927 18:36:43.183439 20279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fedf21779c549529ad5ce8c9476073f" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" is_pre_election: true
I20260927 18:36:43.183545 20411 raft_consensus.cc:2454] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 0.
I20260927 18:36:43.183610 20279 raft_consensus.cc:2454] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 0.
I20260927 18:36:43.183809 20075 leader_election.cc:302] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79, d4d84a8c1eb74614bc512ecd33b5aedb; no voters:
I20260927 18:36:43.183924 20192 raft_consensus.cc:2781] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:36:43.183976 20192 raft_consensus.cc:482] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:36:43.183998 20192 raft_consensus.cc:3037] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:43.184643 20192 raft_consensus.cc:504] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.184762 20192 leader_election.cc:288] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 election: Requested vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
I20260927 18:36:43.184914 20279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fedf21779c549529ad5ce8c9476073f" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb"
I20260927 18:36:43.184930 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fedf21779c549529ad5ce8c9476073f" candidate_uuid: "5788c4cecce6416aad694994dec6ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10847f067c4a42448493cdbdadb85eb7"
I20260927 18:36:43.185002 20411 raft_consensus.cc:3037] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:43.185006 20279 raft_consensus.cc:3037] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:36:43.185674 20411 raft_consensus.cc:2454] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 1.
I20260927 18:36:43.185689 20279 raft_consensus.cc:2454] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5788c4cecce6416aad694994dec6ea79 in term 1.
I20260927 18:36:43.185828 20075 leader_election.cc:302] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79, d4d84a8c1eb74614bc512ecd33b5aedb; no voters:
I20260927 18:36:43.185922 20192 raft_consensus.cc:2781] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:36:43.186065 20192 raft_consensus.cc:686] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 LEADER]: Becoming Leader. State: Replica: 5788c4cecce6416aad694994dec6ea79, State: Running, Role: LEADER
I20260927 18:36:43.186121 20192 consensus_queue.cc:237] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.186949 19997 catalog_manager.cc:5495] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5788c4cecce6416aad694994dec6ea79 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "5788c4cecce6416aad694994dec6ea79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } health_report { overall_health: HEALTHY } } }
I20260927 18:36:43.198848 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.196851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.5.151.193:33997)
0927 18:36:43.196904 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:43.196906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:43.196920 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:36:43.196947 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:43.196948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:43.196961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:43.197021 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:36:43.197111 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.197570 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.198196 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:43.198384 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:43.198430 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:43.198595 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:36:43.198688 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:36:43.198740 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:43.198780 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 18:36:43.198796 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:36:43.200305 20411 raft_consensus.cc:1261] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Refusing update from remote peer 5788c4cecce6416aad694994dec6ea79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:43.200305 20279 raft_consensus.cc:1261] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Refusing update from remote peer 5788c4cecce6416aad694994dec6ea79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:36:43.200663 20192 consensus_queue.cc:1035] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:43.200819 20473 consensus_queue.cc:1035] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:36:43.203673 20476 mvcc.cc:205] Tried to move back new op lower bound from 7334028096305713152 to 7334028096250937344. Current Snapshot: MvccSnapshot[applied={T|T < 7334028096305713152 or (T in {7334028096305713152})}]
I20260927 18:36:43.204839 19997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:43.204936 19997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:36:43.205067 20477 mvcc.cc:205] Tried to move back new op lower bound from 7334028096305713152 to 7334028096250937344. Current Snapshot: MvccSnapshot[applied={T|T < 7334028096305713152}]
I20260927 18:36:43.205696 19997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Sending DeleteTablet for 3 replicas of tablet 9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:43.206027 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 20037
I20260927 18:36:43.206099 20391 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fedf21779c549529ad5ce8c9476073f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:43 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:38162
I20260927 18:36:43.207067 20484 tablet_replica.cc:330] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: stopping tablet replica
I20260927 18:36:43.207165 20484 raft_consensus.cc:2229] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:43.207255 20484 raft_consensus.cc:2258] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:43.207613 20484 ts_tablet_manager.cc:1905] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:43.208320 20259 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fedf21779c549529ad5ce8c9476073f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:36:43 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50178
I20260927 18:36:43.208545 20485 tablet_replica.cc:330] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: stopping tablet replica
I20260927 18:36:43.208638 20485 raft_consensus.cc:2229] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:36:43.208734 20485 raft_consensus.cc:2258] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:36:43.209085 20484 ts_tablet_manager.cc:1918] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:43.209100 20485 ts_tablet_manager.cc:1905] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:36:43.209167 20484 log.cc:1196] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:43.209419 20484 ts_tablet_manager.cc:1939] T 9fedf21779c549529ad5ce8c9476073f P 10847f067c4a42448493cdbdadb85eb7: Deleting consensus metadata
I20260927 18:36:43.209848 19985 catalog_manager.cc:4841] TS 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701): tablet 9fedf21779c549529ad5ce8c9476073f (table test-table [id=a58981885f94437da856c5778f27807a]) successfully deleted
I20260927 18:36:43.210857 20485 ts_tablet_manager.cc:1918] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:36:43.210925 20485 log.cc:1196] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:43.211129 20485 ts_tablet_manager.cc:1939] T 9fedf21779c549529ad5ce8c9476073f P d4d84a8c1eb74614bc512ecd33b5aedb: Deleting consensus metadata
I20260927 18:36:43.211495 19985 catalog_manager.cc:4841] TS d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031): tablet 9fedf21779c549529ad5ce8c9476073f (table test-table [id=a58981885f94437da856c5778f27807a]) successfully deleted
W20260927 18:36:43.212500 19985 connection.cc:535] client connection to 127.5.151.193:33997 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 18:36:43.212607 19985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:36:43.212847 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 20191
I20260927 18:36:43.213469 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.212972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:33997 (local address 127.0.0.1:40934)
0927 18:36:43.213389 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.213428 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396}
W20260927 18:36:43.213599 19985 catalog_manager.cc:4584] TS 5788c4cecce6416aad694994dec6ea79 (127.5.151.193:33997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fedf21779c549529ad5ce8c9476073f: Network error: Client connection negotiation failed: client connection to 127.5.151.193:33997: connect: Connection refused (error 111)
I20260927 18:36:43.217617 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 20328
I20260927 18:36:43.221947 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 19965
I20260927 18:36:43.227123 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32911
--webserver_interface=127.5.151.254
--webserver_port=41647
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:43.305920 20487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:43.306102 20487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:43.306123 20487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:43.307170 20487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:43.307202 20487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:43.307216 20487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:43.307228 20487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:43.307240 20487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:43.308491 20487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=41647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:43.308655 20487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:43.308800 20487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:36:43.313776 20487 server_base.cc:1034] running on GCE node
W20260927 18:36:43.313750 20492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.313786 20495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.313759 20493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:43.314172 20487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:43.314633 20487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:43.315794 20487 hybrid_clock.cc:648] HybridClock initialized: now 1790534203315775 us; error 39 us; skew 500 ppm
Sep 27 18:36:43 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:43.318202 20487 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:43.318722 20487 webserver.cc:460] Webserver started at http://127.5.151.254:41647/ using document root <none> and password file <none>
I20260927 18:36:43.318948 20487 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:43.318991 20487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:43.320298 20487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:36:43.320953 20502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.321111 20487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:36:43.321208 20487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "129b6a8a23af4e599e3e0077c2eba34a"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "77c1f2310ec900c27b653c58d5b81159"
server_key_iv: "07e62b93a6054da6e8796c956c847b56"
server_key_version: "encryptionkey@0"
I20260927 18:36:43.321631 20487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:43.332662 20487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:43.334959 20487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:43.335112 20487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:43.338964 20487 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:32911
I20260927 18:36:43.338991 20554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:32911 every 8 connection(s)
I20260927 18:36:43.339357 20487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:36:43.340794 20555 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:36:43.341315 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 20487
I20260927 18:36:43.341499 20555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Bootstrap starting.
I20260927 18:36:43.341599 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:33997
--local_ip_for_outbound_sockets=127.5.151.193
--tserver_master_addrs=127.5.151.254:32911
--webserver_port=39311
--webserver_interface=127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:43.349591 20555 log.cc:824] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:43.351792 20555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:36:43.352098 20555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a: Bootstrap complete.
I20260927 18:36:43.354247 20555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:43.354471 20555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129b6a8a23af4e599e3e0077c2eba34a, State: Initialized, Role: FOLLOWER
I20260927 18:36:43.354661 20555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:43.354741 20555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:36:43.354790 20555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:36:43.354835 20555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:36:43.355756 20555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:43.355837 20555 leader_election.cc:302] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129b6a8a23af4e599e3e0077c2eba34a; no voters:
I20260927 18:36:43.356032 20555 leader_election.cc:288] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:36:43.356067 20559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:36:43.356179 20559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [term 3 LEADER]: Becoming Leader. State: Replica: 129b6a8a23af4e599e3e0077c2eba34a, State: Running, Role: LEADER
I20260927 18:36:43.356297 20555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:36:43.356273 20559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } }
I20260927 18:36:43.356557 20559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129b6a8a23af4e599e3e0077c2eba34a. Latest consensus state: current_term: 3 leader_uuid: "129b6a8a23af4e599e3e0077c2eba34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } } }
I20260927 18:36:43.356632 20559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:43.356562 20560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "129b6a8a23af4e599e3e0077c2eba34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129b6a8a23af4e599e3e0077c2eba34a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32911 } } }
I20260927 18:36:43.356783 20560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129b6a8a23af4e599e3e0077c2eba34a [sys.catalog]: This master's current role is: LEADER
I20260927 18:36:43.356815 20564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:36:43.357703 20564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:36:43.357827 20564 catalog_manager.cc:1259] Loaded cluster ID: e85e822c58cd440fa267bb6e1332e463
I20260927 18:36:43.358011 20564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 18:36:43.358283 20564 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2d7e3cf9ad google::LogMessage::Fail() at ??:0
@ 0x7f2d7e3d288a google::LogMessage::SendToLog() at ??:0
@ 0x7f2d7e3cf4de google::LogMessage::Flush() at ??:0
@ 0x7f2d7e3d0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2d862f00f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2d862f98b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2d8029d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2d80292954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2d810f66db start_thread at ??:0
@ 0x7f2d7cecd61f clone at ??:0
W20260927 18:36:43.447038 20557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:43.447224 20557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:43.447244 20557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:43.448354 20557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:43.448398 20557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:43.448423 20557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:36:43.449764 20557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:33997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=39311
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:43.449992 20557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:43.450209 20557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:43.454936 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.454928 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.454928 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:43.455294 20557 server_base.cc:1034] running on GCE node
I20260927 18:36:43.455514 20557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:43.455751 20557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:43.456911 20557 hybrid_clock.cc:648] HybridClock initialized: now 1790534203456896 us; error 30 us; skew 500 ppm
Sep 27 18:36:43 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:43.459667 20557 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:36:43.460218 20557 webserver.cc:460] Webserver started at http://127.5.151.193:39311/ using document root <none> and password file <none>
I20260927 18:36:43.460469 20557 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:43.460533 20557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:43.462155 20557 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:36:43.462961 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.463152 20557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 18:36:43.463239 20557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "5788c4cecce6416aad694994dec6ea79"
format_stamp: "Formatted at 2026-09-27 18:36:41 on dist-test-slave-19nl"
server_key: "086520a23b4c11a55a5b11cb4428d983"
server_key_iv: "b157f56643633d7689777250aa2ae555"
server_key_version: "encryptionkey@0"
I20260927 18:36:43.463606 20557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:43.473564 20557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:43.475667 20557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:43.475811 20557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:43.476011 20557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:43.476451 20608 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:36:43.478325 20557 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:36:43.478379 20557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 18:36:43.478425 20557 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:36:43.479039 20557 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:36:43.479086 20557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 18:36:43.479142 20608 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Bootstrap starting.
I20260927 18:36:43.485716 20557 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:33997
I20260927 18:36:43.485755 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:33997 every 8 connection(s)
I20260927 18:36:43.486100 20557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:36:43.486510 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 20557
I20260927 18:36:43.486806 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:35031
--local_ip_for_outbound_sockets=127.5.151.194
--tserver_master_addrs=127.5.151.254:32911
--webserver_port=45367
--webserver_interface=127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:36:43.489396 20608 log.cc:824] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Log is configured to *not* fsync() on all Append() calls
I20260927 18:36:43.491521 20608 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:36:43.491773 20608 tablet_bootstrap.cc:492] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Bootstrap complete.
I20260927 18:36:43.492702 20608 ts_tablet_manager.cc:1397] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260927 18:36:43.493643 20608 raft_consensus.cc:348] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.493881 20608 raft_consensus.cc:729] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5788c4cecce6416aad694994dec6ea79, State: Initialized, Role: FOLLOWER
I20260927 18:36:43.494020 20608 consensus_queue.cc:260] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:43.494242 20608 ts_tablet_manager.cc:1428] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
W20260927 18:36:43.585840 20719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:36:43.488363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:60023)
0927 18:36:43.488531 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.488572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:43.489016 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:43.585630 (+ 96614us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
W20260927 18:36:43.586109 20716 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:32911 (0 consecutive failures): Network error: Failed to ping master at 127.5.151.254:32911: Client connection negotiation failed: client connection to 127.5.151.254:32911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:36:43.586479 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.586309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:35473)
0927 18:36:43.586395 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.586450 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 18:36:43.586884 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.586752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:54947)
0927 18:36:43.586833 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.586862 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:36:43.587009 20716 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:36:43.588470 20718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:43.588652 20718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:43.588706 20718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:43.589860 20718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:43.589898 20718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:43.589952 20718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:36:43.591463 20718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:35031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=45367
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:43.591694 20718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:43.591919 20718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:43.596813 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.596810 20726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.596863 20727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:43.597009 20718 server_base.cc:1034] running on GCE node
I20260927 18:36:43.597198 20718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:43.597402 20718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:43.598624 20718 hybrid_clock.cc:648] HybridClock initialized: now 1790534203598603 us; error 36 us; skew 500 ppm
Sep 27 18:36:43 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:43.600932 20718 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:36:43.601375 20718 webserver.cc:460] Webserver started at http://127.5.151.194:45367/ using document root <none> and password file <none>
I20260927 18:36:43.601594 20718 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:43.601640 20718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:43.602967 20718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:36:43.603489 20736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.603662 20718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:36:43.603754 20718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "d4d84a8c1eb74614bc512ecd33b5aedb"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "23fc55c3dc6814b9ecc198ca282969e2"
server_key_iv: "59eb45632bb70a2d44355417f14543e0"
server_key_version: "encryptionkey@0"
I20260927 18:36:43.604064 20718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:43.617028 20718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:43.618743 20718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:43.618852 20718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:43.619025 20718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:43.619320 20718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:43.619357 20718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.619400 20718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:43.619426 20718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.625182 20718 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:35031
I20260927 18:36:43.625231 20849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:35031 every 8 connection(s)
I20260927 18:36:43.625552 20718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:36:43.626724 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.626268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:57009)
0927 18:36:43.626416 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.626511 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260927 18:36:43.627103 20850 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:32911 (0 consecutive failures): Network error: Failed to ping master at 127.5.151.254:32911: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
I20260927 18:36:43.627372 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.627287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:55429)
0927 18:36:43.627331 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.627357 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:36:43.627665 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.627580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:48307)
0927 18:36:43.627625 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.627647 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:36:43.627799 20850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:36:43.634321 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 20718
I20260927 18:36:43.634644 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:44701
--local_ip_for_outbound_sockets=127.5.151.195
--tserver_master_addrs=127.5.151.254:32911
--webserver_port=34779
--webserver_interface=127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:43.714882 20853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:43.715077 20853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:43.715101 20853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:43.716173 20853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:43.716204 20853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:43.716230 20853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:36:43.717576 20853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:44701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=34779
--tserver_master_addrs=127.5.151.254:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:43.717741 20853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:43.717911 20853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:43.722712 20858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.722723 20861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:43.722730 20859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:43.723067 20853 server_base.cc:1034] running on GCE node
I20260927 18:36:43.723248 20853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:43.723443 20853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:43.724563 20853 hybrid_clock.cc:648] HybridClock initialized: now 1790534203724553 us; error 27 us; skew 500 ppm
Sep 27 18:36:43 dist-test-slave-19nl krb5kdc[19423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:43.726940 20853 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:36:43.727303 20853 webserver.cc:460] Webserver started at http://127.5.151.195:34779/ using document root <none> and password file <none>
I20260927 18:36:43.727479 20853 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:43.727519 20853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:43.728714 20853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:36:43.729279 20868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.729445 20853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.729547 20853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "10847f067c4a42448493cdbdadb85eb7"
format_stamp: "Formatted at 2026-09-27 18:36:42 on dist-test-slave-19nl"
server_key: "1a0845f57f7bec4b9887d2ff29068018"
server_key_iv: "0ae59c150dd07ba964fbfefdfba0fbbf"
server_key_version: "encryptionkey@0"
I20260927 18:36:43.729868 20853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:36:43.745435 20853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:36:43.747225 20853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:36:43.747383 20853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:36:43.747604 20853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:36:43.747902 20853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:36:43.747954 20853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.747997 20853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:36:43.748036 20853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:43.753578 20853 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:44701
I20260927 18:36:43.753652 20981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:44701 every 8 connection(s)
I20260927 18:36:43.753994 20853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:36:43.755053 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.754711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:58085)
0927 18:36:43.754846 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.754917 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
W20260927 18:36:43.755460 20982 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:32911 (0 consecutive failures): Network error: Failed to ping master at 127.5.151.254:32911: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
I20260927 18:36:43.755702 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.755621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:50769)
0927 18:36:43.755662 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.755686 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 18:36:43.755949 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:43.755896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:50341)
0927 18:36:43.755919 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:43.755937 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 18:36:43.756062 20982 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:36:43.759193 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 20853
I20260927 18:36:44.587489 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.587295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:38977)
0927 18:36:44.587400 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.587432 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260927 18:36:44.628365 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.628090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:34443)
0927 18:36:44.628288 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.628322 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
I20260927 18:36:44.664021 20988 raft_consensus.cc:482] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:44.664130 20988 raft_consensus.cc:504] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:44.664398 20988 leader_election.cc:288] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
Sep 27 18:36:44 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.195@KRBTEST.COM
Sep 27 18:36:44 dist-test-slave-19nl krb5kdc[19423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.194@KRBTEST.COM
I20260927 18:36:44.674788 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.664886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:44223 (local address 127.5.151.195:44701)
0927 18:36:44.665025 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:44.665033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:44.665054 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:36:44.665431 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:44.665433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:44.665506 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:44.665637 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:44.666469 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.667672 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.668520 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.668937 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.669045 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:44.671179 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:44.671184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:44.671184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:44.671196 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:44.673083 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673553 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:44.673556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:44.673557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:44.673667 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673933 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:44.673934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:44.673934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:44.674173 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:44.674290 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:44.674569 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 18:36:44.674696 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260927 18:36:44.674822 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.664681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:48459 (local address 127.5.151.194:35031)
0927 18:36:44.664800 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:36:44.664835 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 18:36:44.664871 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:36:44.665346 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:36:44.665349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:36:44.665392 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:36:44.665508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:36:44.666435 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.667511 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.668486 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.668947 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.669066 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:44.671540 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:36:44.671544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:36:44.671544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:36:44.671554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:36:44.673204 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673609 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:44.673612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:44.673612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:44.673711 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673956 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:36:44.673957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:36:44.673957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:36:44.674172 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:36:44.674270 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 18:36:44.674565 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 18:36:44.674713 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":47,"threads_started":1}
I20260927 18:36:44.674830 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.664560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:35031 (local address 127.5.151.193:48459)
0927 18:36:44.664613 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.664625 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:44.664680 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:44.665576 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:44.665579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:44.666012 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:44.666303 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:44.666310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.667599 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.667601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:44.668401 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:44.668404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.669044 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.669045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:44.669148 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:44.669773 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:44.669778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:44.671420 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:44.673291 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:44.673292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:44.673526 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:44.673794 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:44.673795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:44.673868 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:44.674251 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:44.674269 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:44.674392 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 18:36:44.674485 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:36:44.674687 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":3}
I20260927 18:36:44.675426 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.664695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:44701 (local address 127.5.151.193:44223)
0927 18:36:44.664804 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.664813 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:36:44.664853 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:36:44.665615 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:36:44.665616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:36:44.666013 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:36:44.666302 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:44.666308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.667781 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.667782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:44.668400 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:36:44.668402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:36:44.669019 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:36:44.669020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:36:44.669141 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:36:44.669619 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:36:44.669627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:36:44.671074 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:36:44.673166 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:44.673175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:44.673477 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:44.673736 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:36:44.673737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:36:44.673737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:36:44.673867 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:36:44.674248 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:36:44.674266 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:36:44.674388 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 18:36:44.674485 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:36:44.674764 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":130,"thread_start_us":58,"threads_started":1}
W20260927 18:36:44.675690 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
W20260927 18:36:44.676252 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:44.676303 20605 leader_election.cc:302] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79; no voters: 10847f067c4a42448493cdbdadb85eb7, d4d84a8c1eb74614bc512ecd33b5aedb
I20260927 18:36:44.676409 20988 raft_consensus.cc:2726] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:36:44.756533 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:44.756349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:53411)
0927 18:36:44.756468 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:44.756503 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260927 18:36:45.588084 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:45.587873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:55927)
0927 18:36:45.588004 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:45.588038 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260927 18:36:45.629076 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:45.628863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:38877)
0927 18:36:45.628997 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:45.629034 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260927 18:36:45.757061 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:45.756896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:43933)
0927 18:36:45.757000 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:45.757029 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20260927 18:36:46.588843 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:46.588602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:38825)
0927 18:36:46.588763 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:46.588796 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260927 18:36:46.629703 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:46.629510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:59101)
0927 18:36:46.629631 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:46.629669 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260927 18:36:46.757606 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:46.757425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:45745)
0927 18:36:46.757537 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:46.757571 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260927 18:36:46.768507 20999 raft_consensus.cc:482] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:46.768584 20999 raft_consensus.cc:504] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:46.768767 20999 leader_election.cc:288] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
W20260927 18:36:46.769173 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
W20260927 18:36:46.769260 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:46.769305 20605 leader_election.cc:302] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79; no voters: 10847f067c4a42448493cdbdadb85eb7, d4d84a8c1eb74614bc512ecd33b5aedb
I20260927 18:36:46.769410 20999 raft_consensus.cc:2726] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:36:47.589550 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:47.589303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:59323)
0927 18:36:47.589470 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:47.589503 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260927 18:36:47.630262 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:47.630079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:60545)
0927 18:36:47.630189 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:47.630222 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260927 18:36:47.758227 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:47.758030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:37939)
0927 18:36:47.758148 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:47.758187 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260927 18:36:48.590219 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:48.590028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:50369)
0927 18:36:48.590143 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:48.590173 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260927 18:36:48.630908 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:48.630729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:33883)
0927 18:36:48.630838 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:48.630872 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260927 18:36:48.758900 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:48.758719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:42023)
0927 18:36:48.758829 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:48.758863 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260927 18:36:48.791594 21006 raft_consensus.cc:482] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:48.791692 21006 raft_consensus.cc:504] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:48.791879 21006 leader_election.cc:288] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
W20260927 18:36:48.792376 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
W20260927 18:36:48.792439 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:48.792455 20605 leader_election.cc:302] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79; no voters: 10847f067c4a42448493cdbdadb85eb7, d4d84a8c1eb74614bc512ecd33b5aedb
I20260927 18:36:48.792551 21006 raft_consensus.cc:2726] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:36:49.590936 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:49.590731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:39529)
0927 18:36:49.590860 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:49.590892 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260927 18:36:49.631457 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:49.631279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:53547)
0927 18:36:49.631393 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:49.631420 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260927 18:36:49.759482 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:49.759288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:52175)
0927 18:36:49.759416 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:49.759447 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260927 18:36:50.591584 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:50.591363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:56421)
0927 18:36:50.591502 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:50.591539 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":57,"threads_started":1}
I20260927 18:36:50.632040 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:50.631865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:50595)
0927 18:36:50.631975 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:50.632004 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260927 18:36:50.760046 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:50.759867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:36837)
0927 18:36:50.759984 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:50.760013 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260927 18:36:51.592433 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:51.592228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:36677)
0927 18:36:51.592356 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:51.592389 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260927 18:36:51.632627 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:51.632432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:38303)
0927 18:36:51.632545 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:51.632588 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260927 18:36:51.760651 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:51.760463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:51709)
0927 18:36:51.760583 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:51.760616 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260927 18:36:51.968271 21016 raft_consensus.cc:482] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:51.968379 21016 raft_consensus.cc:504] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:51.968503 21016 leader_election.cc:288] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
W20260927 18:36:51.968951 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
W20260927 18:36:51.969041 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:51.969060 20605 leader_election.cc:302] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79; no voters: 10847f067c4a42448493cdbdadb85eb7, d4d84a8c1eb74614bc512ecd33b5aedb
I20260927 18:36:51.969180 21016 raft_consensus.cc:2726] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:36:52.593091 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:52.592861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:48763)
0927 18:36:52.593005 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:52.593046 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260927 18:36:52.633288 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:52.633088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:51345)
0927 18:36:52.633212 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:52.633246 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260927 18:36:52.761330 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:52.761113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:33837)
0927 18:36:52.761244 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:52.761290 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260927 18:36:53.593797 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:53.593598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:58437)
0927 18:36:53.593716 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:53.593753 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260927 18:36:53.633989 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:53.633775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:46667)
0927 18:36:53.633907 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:53.633943 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260927 18:36:53.761960 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:53.761759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:43953)
0927 18:36:53.761880 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:53.761918 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260927 18:36:54.594491 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:54.594290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:54619)
0927 18:36:54.594413 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:54.594447 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260927 18:36:54.634696 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:54.634455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:59435)
0927 18:36:54.634619 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:54.634652 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260927 18:36:54.762638 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:54.762379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:53925)
0927 18:36:54.762522 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:54.762599 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260927 18:36:55.507943 21026 raft_consensus.cc:482] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:36:55.508051 21026 raft_consensus.cc:504] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d84a8c1eb74614bc512ecd33b5aedb" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 35031 } } peers { permanent_uuid: "10847f067c4a42448493cdbdadb85eb7" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 44701 } } peers { permanent_uuid: "5788c4cecce6416aad694994dec6ea79" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 33997 } }
I20260927 18:36:55.508237 21026 leader_election.cc:288] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031), 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701)
W20260927 18:36:55.508639 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4d84a8c1eb74614bc512ecd33b5aedb (127.5.151.194:35031): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
W20260927 18:36:55.508730 20605 leader_election.cc:341] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10847f067c4a42448493cdbdadb85eb7 (127.5.151.195:44701): Not found: Tablet not found: 9fedf21779c549529ad5ce8c9476073f
I20260927 18:36:55.508780 20605 leader_election.cc:302] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5788c4cecce6416aad694994dec6ea79; no voters: 10847f067c4a42448493cdbdadb85eb7, d4d84a8c1eb74614bc512ecd33b5aedb
I20260927 18:36:55.508867 21026 raft_consensus.cc:2726] T 9fedf21779c549529ad5ce8c9476073f P 5788c4cecce6416aad694994dec6ea79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:36:55.595259 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:55.594986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:37415)
0927 18:36:55.595157 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:55.595197 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260927 18:36:55.635313 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:55.635106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:35597)
0927 18:36:55.635233 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:55.635266 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260927 18:36:55.763223 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:55.763021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:45405)
0927 18:36:55.763148 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:55.763182 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260927 18:36:56.595855 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:56.595682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:33443)
0927 18:36:56.595786 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:56.595813 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260927 18:36:56.635916 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:56.635718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:58435)
0927 18:36:56.635842 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:56.635873 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260927 18:36:56.763854 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:56.763681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:42579)
0927 18:36:56.763786 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:56.763815 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260927 18:36:57.596510 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:57.596300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:41481)
0927 18:36:57.596428 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:57.596466 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260927 18:36:57.636516 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:57.636324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:51075)
0927 18:36:57.636439 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:57.636473 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260927 18:36:57.764518 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:57.764299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:45021)
0927 18:36:57.764422 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:57.764457 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260927 18:36:58.597256 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:58.597003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.193:50307)
0927 18:36:58.597145 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:58.597196 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260927 18:36:58.637154 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:58.636946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.194:43095)
0927 18:36:58.637077 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:58.637111 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260927 18:36:58.763213 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 20557
I20260927 18:36:58.765153 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:36:58.764978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32911 (local address 127.5.151.195:45651)
0927 18:36:58.765089 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:36:58.765119 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260927 18:36:58.768129 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 20718
I20260927 18:36:58.772246 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 20853
2026-09-27T18:36:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18222 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:36:58 dist-test-slave-19nl krb5kdc[21041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:36:58 dist-test-slave-19nl krb5kdc[21041](info): set up 2 sockets
Sep 27 18:36:58 dist-test-slave-19nl krb5kdc[21041](info): commencing operation
krb5kdc: starting...
W20260927 18:36:59.786257 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:36:59 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:36:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:36:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:36:59.896569 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35259
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:36:59.973558 21054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:36:59.973742 21054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:36:59.973762 21054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:36:59.974900 21054 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:36:59.974937 21054 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:36:59.974949 21054 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:36:59.974961 21054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:36:59.975018 21054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:36:59.975034 21054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:36:59.975059 21054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:36:59.975086 21054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:36:59.976454 21054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:35259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:36:59.976717 21054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:36:59.976939 21054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:36:59.981961 21060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:59.982064 21054 server_base.cc:1034] running on GCE node
W20260927 18:36:59.981949 21062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:36:59.982120 21059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:36:59.982523 21054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:36:59.982803 21054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:36:59.983968 21054 hybrid_clock.cc:648] HybridClock initialized: now 1790534219983944 us; error 49 us; skew 500 ppm
Sep 27 18:36:59 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:36:59.986775 21054 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:36:59.987277 21054 webserver.cc:460] Webserver started at http://127.5.151.254:42759/ using document root <none> and password file <none>
I20260927 18:36:59.987517 21054 fs_manager.cc:359] Metadata directory not provided
I20260927 18:36:59.987579 21054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:36:59.987713 21054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:36:59.988911 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "76f6f8f8366347cebd5449f8e071c16a"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-19nl"
server_key: "f35ecc07c0d34fc5337df2bfa54ef181"
server_key_iv: "8b7688546f1ccdbd204197b2672f6943"
server_key_version: "encryptionkey@0"
I20260927 18:36:59.989323 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "76f6f8f8366347cebd5449f8e071c16a"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-19nl"
server_key: "f35ecc07c0d34fc5337df2bfa54ef181"
server_key_iv: "8b7688546f1ccdbd204197b2672f6943"
server_key_version: "encryptionkey@0"
I20260927 18:36:59.991147 21054 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 18:36:59.991866 21069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:36:59.992036 21054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:36:59.992132 21054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "76f6f8f8366347cebd5449f8e071c16a"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-19nl"
server_key: "f35ecc07c0d34fc5337df2bfa54ef181"
server_key_iv: "8b7688546f1ccdbd204197b2672f6943"
server_key_version: "encryptionkey@0"
I20260927 18:36:59.992223 21054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:00.013917 21054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:00.015602 21054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:00.015750 21054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:00.019522 21054 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:35259
I20260927 18:37:00.019557 21121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:35259 every 8 connection(s)
I20260927 18:37:00.019886 21054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:00.020692 21122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:00.021000 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21054
I20260927 18:37:00.021100 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:37:00.021301 5727 external_mini_cluster.cc:1428] Setting key d974e62deaf965ef1957d8958f64dbab
I20260927 18:37:00.023892 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Bootstrap starting.
I20260927 18:37:00.024536 21122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:00.024825 21122 log.cc:824] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:00.025584 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: No bootstrap required, opened a new log
Sep 27 18:37:00 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:00.028003 21122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:00.028122 21122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:00.028162 21122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f6f8f8366347cebd5449f8e071c16a, State: Initialized, Role: FOLLOWER
I20260927 18:37:00.028282 21122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:00.028348 21122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:00.028368 21122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:00.028422 21122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:00.028930 21122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:00.029040 21122 leader_election.cc:302] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f6f8f8366347cebd5449f8e071c16a; no voters:
I20260927 18:37:00.029177 21122 leader_election.cc:288] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:00.029206 21127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:00.029300 21127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 LEADER]: Becoming Leader. State: Replica: 76f6f8f8366347cebd5449f8e071c16a, State: Running, Role: LEADER
I20260927 18:37:00.029381 21127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:00.029413 21122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:00.029659 21127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f6f8f8366347cebd5449f8e071c16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } } }
I20260927 18:37:00.029731 21127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:00.029852 21127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f6f8f8366347cebd5449f8e071c16a. Latest consensus state: current_term: 1 leader_uuid: "76f6f8f8366347cebd5449f8e071c16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } } }
I20260927 18:37:00.029914 21127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:00.030530 21131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:00.031023 21131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:00.031769 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.021761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.5.151.254:35259)
0927 18:37:00.021928 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.021939 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.021962 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:37:00.022383 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.022389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.022431 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.022537 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:00.023283 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.024133 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.025107 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.025537 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.025610 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.028122 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:00.028126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:00.028127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:00.028137 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:00.029980 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.030373 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.030376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.030377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.030486 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.030751 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.030752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.030753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.031036 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:00.031199 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.031456 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.031552 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 18:37:00.034917 21131 catalog_manager.cc:1347] Generated new cluster ID: 64cac6bf084046e5be17502c5d402b0d
I20260927 18:37:00.034981 21131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:00.042253 21131 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:00.042912 21131 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:00.045666 21131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Generated new TSK 0
I20260927 18:37:00.045830 21131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:37:00.073073 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35259
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:00.148404 21148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:00.148574 21148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:00.148594 21148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:00.149564 21148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:00.149595 21148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:00.149617 21148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:37:00.150844 21148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:00.150997 21148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:00.151154 21148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:00.155753 21156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.155779 21154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.155745 21153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:00.155915 21148 server_base.cc:1034] running on GCE node
I20260927 18:37:00.156122 21148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:00.156390 21148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:00.157550 21148 hybrid_clock.cc:648] HybridClock initialized: now 1790534220157539 us; error 31 us; skew 500 ppm
Sep 27 18:37:00 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:00.159868 21148 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:37:00.160333 21148 webserver.cc:460] Webserver started at http://127.5.151.193:42041/ using document root <none> and password file <none>
I20260927 18:37:00.160562 21148 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:00.160624 21148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:00.160758 21148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:00.161733 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "c960fca8e06c48d5b08f7a09619846a7"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "60188db7e241fc5d3218df03983f75d1"
server_key_iv: "d91cc18728ed7363599908db1b3ea48a"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.162082 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "c960fca8e06c48d5b08f7a09619846a7"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "60188db7e241fc5d3218df03983f75d1"
server_key_iv: "d91cc18728ed7363599908db1b3ea48a"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.163450 21148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 18:37:00.164117 21163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.164290 21148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:00.164382 21148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "c960fca8e06c48d5b08f7a09619846a7"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "60188db7e241fc5d3218df03983f75d1"
server_key_iv: "d91cc18728ed7363599908db1b3ea48a"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.164458 21148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:00.185308 21148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:00.186988 21148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:00.187129 21148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:00.187325 21148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:00.187608 21148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:00.187669 21148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.187705 21148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:00.187736 21148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.192948 21148 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:36705
I20260927 18:37:00.192983 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:36705 every 8 connection(s)
I20260927 18:37:00.193284 21148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:00.197055 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21148
I20260927 18:37:00.197158 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:00.197403 5727 external_mini_cluster.cc:1428] Setting key 4a32a79dc86bd6771832f529b2155ffb
Sep 27 18:37:00 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:00.203842 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.193939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:44441 (local address 127.5.151.254:35259)
0927 18:37:00.193982 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.193984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.194199 (+ 215us) server_negotiation.cc:407] Connection header received
0927 18:37:00.194682 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.194682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.194691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.194742 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:00.195584 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.196139 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.197056 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.197439 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.197520 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.200944 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:00.200946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:00.200947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:00.200954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:00.202421 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.202937 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.202937 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.202938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.202986 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.203261 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.203262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.203262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.203354 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:00.203398 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.203720 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.203783 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:37:00.204074 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.193891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:44441)
0927 18:37:00.194033 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.194101 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.194524 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.194795 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.194798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.195320 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:00.195506 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.195510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.196224 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.196225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.196968 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.196970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.197547 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.197549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.197664 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.199361 (+ 1697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:00.199368 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:00.200852 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:00.202502 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:00.202510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:00.202512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:00.202855 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:00.203052 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:00.203053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:00.203053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:00.203192 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:00.203413 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:00.203420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:00.203574 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.203692 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.203860 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260927 18:37:00.204653 21277 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35259
I20260927 18:37:00.204763 21277 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:00.204928 21277 heartbeater.cc:511] Master 127.5.151.254:35259 requested a full tablet report, sending...
I20260927 18:37:00.205425 21086 ts_manager.cc:194] Registered new tserver with Master: c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705)
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
I20260927 18:37:00.206045 21086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:44441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:37:00.220399 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35259
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:00.295887 21282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:00.296123 21282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:00.296145 21282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:00.297186 21282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:00.297219 21282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:00.297241 21282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:37:00.298602 21282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:00.298763 21282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:00.298909 21282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:00.303485 21287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.303399 21288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.303398 21290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:00.303562 21282 server_base.cc:1034] running on GCE node
I20260927 18:37:00.303815 21282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:00.304064 21282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:00.305223 21282 hybrid_clock.cc:648] HybridClock initialized: now 1790534220305212 us; error 27 us; skew 500 ppm
Sep 27 18:37:00 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:00.307564 21282 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:37:00.307945 21282 webserver.cc:460] Webserver started at http://127.5.151.194:33155/ using document root <none> and password file <none>
I20260927 18:37:00.308166 21282 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:00.308230 21282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:00.308373 21282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:00.309325 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "d6338dc9628a4ae9adcd5a44c4ca6875"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "3cfc5d9259808647f5b1e2383dfb9480"
server_key_iv: "f8201629c9f7de487124f86a1c1de84e"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.309664 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "d6338dc9628a4ae9adcd5a44c4ca6875"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "3cfc5d9259808647f5b1e2383dfb9480"
server_key_iv: "f8201629c9f7de487124f86a1c1de84e"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.311022 21282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:37:00.311782 21297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.311931 21282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:00.312026 21282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "d6338dc9628a4ae9adcd5a44c4ca6875"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "3cfc5d9259808647f5b1e2383dfb9480"
server_key_iv: "f8201629c9f7de487124f86a1c1de84e"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.312114 21282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:00.332479 21282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:00.334810 21282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:00.334966 21282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:00.335191 21282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:00.335511 21282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:00.335561 21282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.335592 21282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:00.335621 21282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.341100 21282 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:33109
I20260927 18:37:00.341161 21410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:33109 every 8 connection(s)
I20260927 18:37:00.341480 21282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:00.344630 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21282
I20260927 18:37:00.344722 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:00.344918 5727 external_mini_cluster.cc:1428] Setting key 16d677b873aaac6ddf9bc81217d1beaa
Sep 27 18:37:00 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:00.350495 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.342153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:56575 (local address 127.5.151.254:35259)
0927 18:37:00.342238 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.342240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.342382 (+ 142us) server_negotiation.cc:407] Connection header received
0927 18:37:00.342835 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.342835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.342844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.342892 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:00.343698 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.344161 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.345305 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.345551 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.345627 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.347829 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:00.347830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:00.347830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:00.347838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:00.349170 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.349578 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.349580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.349580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.349636 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.349875 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.349876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.349876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.349991 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:00.350066 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.350334 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.350410 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:37:00.350703 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.342105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:56575)
0927 18:37:00.342269 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.342302 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.342754 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.342942 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.342944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.343453 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:00.343631 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.343635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.344240 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.344241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.345210 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.345212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.345653 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.345654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.345804 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.346228 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:00.346237 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:00.347734 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:00.349269 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:00.349276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:00.349278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:00.349489 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:00.349706 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:00.349707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:00.349707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:00.349810 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:00.350058 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:00.350064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:00.350200 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.350300 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.350444 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":108,"threads_started":1}
I20260927 18:37:00.351287 21411 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35259
I20260927 18:37:00.351361 21411 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:00.351538 21411 heartbeater.cc:511] Master 127.5.151.254:35259 requested a full tablet report, sending...
I20260927 18:37:00.352000 21086 ts_manager.cc:194] Registered new tserver with Master: d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109)
I20260927 18:37:00.352500 21086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:56575
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:37:00.369158 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35259
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:00.443919 21416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:00.444096 21416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:00.444118 21416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:00.445142 21416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:00.445173 21416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:00.445197 21416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:37:00.446496 21416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:00.446672 21416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:00.446826 21416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:37:00.451390 21416 server_base.cc:1034] running on GCE node
W20260927 18:37:00.451366 21422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.451364 21424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:00.451360 21421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:00.451717 21416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:00.451964 21416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:00.453117 21416 hybrid_clock.cc:648] HybridClock initialized: now 1790534220453105 us; error 24 us; skew 500 ppm
Sep 27 18:37:00 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:00.455415 21416 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:37:00.455879 21416 webserver.cc:460] Webserver started at http://127.5.151.195:33285/ using document root <none> and password file <none>
I20260927 18:37:00.456104 21416 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:00.456185 21416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:00.456332 21416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:00.457266 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "26604490b4ba43f4a693ea98bb2a1390"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "87ce29dc46c75237d475e56abee5555d"
server_key_iv: "4c99cbe7c2190cfc25670aabd51f7f0e"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.457590 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "26604490b4ba43f4a693ea98bb2a1390"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "87ce29dc46c75237d475e56abee5555d"
server_key_iv: "4c99cbe7c2190cfc25670aabd51f7f0e"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.458871 21416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:37:00.459581 21431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.459779 21416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:00.459873 21416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "26604490b4ba43f4a693ea98bb2a1390"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "87ce29dc46c75237d475e56abee5555d"
server_key_iv: "4c99cbe7c2190cfc25670aabd51f7f0e"
server_key_version: "encryptionkey@0"
I20260927 18:37:00.459946 21416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:00.468510 21416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:00.470150 21416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:00.470294 21416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:00.470491 21416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:00.470824 21416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:00.470877 21416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.470908 21416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:00.470937 21416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:00.476454 21416 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:37651
I20260927 18:37:00.476488 21544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:37651 every 8 connection(s)
I20260927 18:37:00.476783 21416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:37:00 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:00.483152 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21416
I20260927 18:37:00.483266 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:00.483445 5727 external_mini_cluster.cc:1428] Setting key ade403f66ced781dfe5fcf4094cf7f77
I20260927 18:37:00.485190 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.477423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:47539 (local address 127.5.151.254:35259)
0927 18:37:00.477463 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.477465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.477637 (+ 172us) server_negotiation.cc:407] Connection header received
0927 18:37:00.478117 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.478119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.478130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.478176 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:00.479070 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.479547 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.480462 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.480662 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.480734 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.482513 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:00.482515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:00.482515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:00.482521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:00.484047 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.484484 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.484485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.484485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.484536 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.484734 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.484735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.484735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.484827 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:00.484875 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.485063 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.485120 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:37:00.485375 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.477378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:47539)
0927 18:37:00.477528 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.477566 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.478008 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.478237 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.478240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.478805 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:00.478966 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.478970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.479636 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.479637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.480377 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.480378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.480740 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.480741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.480852 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.481235 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:00.481243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:00.482430 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:00.484158 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:00.484164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:00.484165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:00.484423 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:00.484593 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:00.484594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:00.484594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:00.484683 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:00.484877 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:00.484881 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:00.484978 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.485046 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.485161 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260927 18:37:00.485819 21545 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35259
I20260927 18:37:00.485886 21545 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:00.486054 21545 heartbeater.cc:511] Master 127.5.151.254:35259 requested a full tablet report, sending...
I20260927 18:37:00.486407 21086 ts_manager.cc:194] Registered new tserver with Master: 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651)
I20260927 18:37:00.486779 21086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:47539
I20260927 18:37:00.495141 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:00.501649 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.496416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.5.151.254:35259)
0927 18:37:00.496464 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.496466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.496584 (+ 118us) server_negotiation.cc:407] Connection header received
0927 18:37:00.496626 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.496626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.496634 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.496681 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:00.497162 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.497578 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.498215 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.498406 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.498460 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.499588 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:00.499589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:00.499590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:00.499595 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:00.500756 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.501067 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.501067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.501067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.501111 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:00.501265 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:00.501266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:00.501266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:00.501380 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:00.501450 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.501538 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.501593 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:37:00.503105 21086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:00.504563 21086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:00.510296 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.506487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36705 (local address 127.0.0.1:50650)
0927 18:37:00.506805 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.506841 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.506906 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.507092 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.507094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.507096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:00.507154 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.507156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.508118 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.508119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.509082 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.509121 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.510023 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.510024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.510100 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.510112 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.510183 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.510225 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":37,"spinlock_wait_cycles":384,"thread_start_us":93,"threads_started":1}
I20260927 18:37:00.510356 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.506631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.5.151.193:36705)
0927 18:37:00.506896 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.506901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.506913 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:37:00.506994 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.506996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.507010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.507089 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:00.507216 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.508020 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.509214 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.509938 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.510010 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.510063 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.510208 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.510253 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":79,"threads_started":1}
I20260927 18:37:00.510680 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.506711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37651 (local address 127.0.0.1:52614)
0927 18:37:00.507015 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.507025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.507064 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.507264 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.507264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.507266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:00.507297 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.507298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.508126 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.508127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.509488 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.509491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.510413 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.510414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.510483 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.510489 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.510568 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.510611 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":23,"thread_start_us":81,"threads_started":1}
I20260927 18:37:00.510807 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.506821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.5.151.195:37651)
0927 18:37:00.507088 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.507092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.507102 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:37:00.507175 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.507177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.507193 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.507268 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:00.507350 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.508026 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.509570 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.510263 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.510327 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.510377 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.510596 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.510637 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":62,"threads_started":1}
I20260927 18:37:00.511129 21211 tablet_service.cc:1467] Processing CreateTablet for tablet ac4475590c9a47a39d5e939e4773a50d (DEFAULT_TABLE table=test-table [id=f9827d49707e493f9e6b56c223a1b996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:00.511353 21479 tablet_service.cc:1467] Processing CreateTablet for tablet ac4475590c9a47a39d5e939e4773a50d (DEFAULT_TABLE table=test-table [id=f9827d49707e493f9e6b56c223a1b996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:00.511473 21211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4475590c9a47a39d5e939e4773a50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:00.511669 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4475590c9a47a39d5e939e4773a50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:00.511822 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.506487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:33109 (local address 127.0.0.1:52624)
0927 18:37:00.506831 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.506843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.506905 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.508385 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.508386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.508389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:00.508431 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.508433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.509625 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.509626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.510681 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.510684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.511582 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.511583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.511625 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.511646 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.511721 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.511760 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260927 18:37:00.511862 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.506629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.5.151.194:33109)
0927 18:37:00.508232 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.508236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.508245 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:37:00.508279 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.508281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.508297 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.508371 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:00.508491 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.509522 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.510837 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.511448 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.511493 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.511550 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.511748 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.511784 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":83,"threads_started":1}
I20260927 18:37:00.512362 21345 tablet_service.cc:1467] Processing CreateTablet for tablet ac4475590c9a47a39d5e939e4773a50d (DEFAULT_TABLE table=test-table [id=f9827d49707e493f9e6b56c223a1b996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:00.512573 21345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4475590c9a47a39d5e939e4773a50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:00.514918 21560 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Bootstrap starting.
I20260927 18:37:00.515110 21559 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Bootstrap starting.
I20260927 18:37:00.515125 21561 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Bootstrap starting.
I20260927 18:37:00.515566 21560 tablet_bootstrap.cc:654] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:00.515764 21559 tablet_bootstrap.cc:654] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:00.515765 21561 tablet_bootstrap.cc:654] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:00.516274 21559 log.cc:824] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:00.516270 21560 log.cc:824] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:00.516413 21561 log.cc:824] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:00.517002 21560 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: No bootstrap required, opened a new log
I20260927 18:37:00.517082 21560 ts_tablet_manager.cc:1397] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:37:00.518525 21560 raft_consensus.cc:348] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.518745 21560 raft_consensus.cc:374] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:00.518795 21560 raft_consensus.cc:729] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26604490b4ba43f4a693ea98bb2a1390, State: Initialized, Role: FOLLOWER
I20260927 18:37:00.518903 21559 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: No bootstrap required, opened a new log
I20260927 18:37:00.518906 21560 consensus_queue.cc:260] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.519016 21559 ts_tablet_manager.cc:1397] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 18:37:00.519186 21560 ts_tablet_manager.cc:1428] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:37:00.519209 21545 heartbeater.cc:503] Master 127.5.151.254:35259 was elected leader, sending a full tablet report...
I20260927 18:37:00.520701 21559 raft_consensus.cc:348] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.520821 21559 raft_consensus.cc:374] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:00.520841 21559 raft_consensus.cc:729] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c960fca8e06c48d5b08f7a09619846a7, State: Initialized, Role: FOLLOWER
I20260927 18:37:00.520936 21559 consensus_queue.cc:260] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.521255 21559 ts_tablet_manager.cc:1428] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:37:00.521288 21277 heartbeater.cc:503] Master 127.5.151.254:35259 was elected leader, sending a full tablet report...
I20260927 18:37:00.521790 21561 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: No bootstrap required, opened a new log
I20260927 18:37:00.521842 21561 ts_tablet_manager.cc:1397] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 18:37:00.523144 21561 raft_consensus.cc:348] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.523272 21561 raft_consensus.cc:374] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:00.523304 21561 raft_consensus.cc:729] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6338dc9628a4ae9adcd5a44c4ca6875, State: Initialized, Role: FOLLOWER
I20260927 18:37:00.523399 21561 consensus_queue.cc:260] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.523654 21561 ts_tablet_manager.cc:1428] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:37:00.523676 21411 heartbeater.cc:503] Master 127.5.151.254:35259 was elected leader, sending a full tablet report...
I20260927 18:37:00.534170 21565 raft_consensus.cc:482] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:00.534273 21565 raft_consensus.cc:504] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.534612 21565 leader_election.cc:288] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109), c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705)
I20260927 18:37:00.537030 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.534778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:59759 (local address 127.5.151.194:33109)
0927 18:37:00.534881 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.534882 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.534889 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:37:00.534909 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.534910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.534922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.534982 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:00.535105 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.535525 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.536360 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.536717 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.536793 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.536801 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.536933 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.536977 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:37:00.537034 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.534718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:33109 (local address 127.5.151.195:59759)
0927 18:37:00.534755 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.534766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.534806 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.534984 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.534985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.534986 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:00.535041 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.535042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.535607 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.535608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.536268 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.536269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.536791 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.536792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.536865 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.536867 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.536929 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.536977 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 18:37:00.537288 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.534826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:58095 (local address 127.5.151.193:36705)
0927 18:37:00.534867 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.534869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.534942 (+ 73us) server_negotiation.cc:407] Connection header received
0927 18:37:00.535004 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.535005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.535014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.535074 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:00.535205 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.535632 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.536500 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.536865 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.536945 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.536952 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.537190 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.537231 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:37:00.537330 21365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4475590c9a47a39d5e939e4773a50d" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" is_pre_election: true
I20260927 18:37:00.537448 21365 raft_consensus.cc:2454] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 0.
I20260927 18:37:00.537508 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.534755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36705 (local address 127.5.151.195:58095)
0927 18:37:00.534866 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.534877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:00.534957 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:00.535075 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:00.535076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:00.535077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:00.535142 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.535143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.535713 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.535714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.536414 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:00.536416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.536945 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.536946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:00.537026 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.537086 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:37:00.537394 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 18:37:00.537434 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260927 18:37:00.537650 21433 leader_election.cc:302] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875; no voters:
I20260927 18:37:00.537752 21565 raft_consensus.cc:2781] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:00.537801 21565 raft_consensus.cc:482] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:00.537823 21565 raft_consensus.cc:3037] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:00.537838 21231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4475590c9a47a39d5e939e4773a50d" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c960fca8e06c48d5b08f7a09619846a7" is_pre_election: true
I20260927 18:37:00.537954 21231 raft_consensus.cc:2454] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 0.
I20260927 18:37:00.538457 21565 raft_consensus.cc:504] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.538591 21565 leader_election.cc:288] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 election: Requested vote from peers d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109), c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705)
I20260927 18:37:00.538741 21365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4475590c9a47a39d5e939e4773a50d" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875"
I20260927 18:37:00.538753 21231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4475590c9a47a39d5e939e4773a50d" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c960fca8e06c48d5b08f7a09619846a7"
I20260927 18:37:00.538818 21365 raft_consensus.cc:3037] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:00.538815 21231 raft_consensus.cc:3037] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:00.539376 21365 raft_consensus.cc:2454] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 1.
I20260927 18:37:00.539489 21231 raft_consensus.cc:2454] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 1.
I20260927 18:37:00.539579 21433 leader_election.cc:302] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875; no voters:
I20260927 18:37:00.539659 21565 raft_consensus.cc:2781] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:00.539805 21565 raft_consensus.cc:686] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 1 LEADER]: Becoming Leader. State: Replica: 26604490b4ba43f4a693ea98bb2a1390, State: Running, Role: LEADER
I20260927 18:37:00.539891 21565 consensus_queue.cc:237] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:00.540575 21086 catalog_manager.cc:5495] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "26604490b4ba43f4a693ea98bb2a1390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } health_report { overall_health: HEALTHY } } }
I20260927 18:37:00.548628 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.546744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.5.151.195:37651)
0927 18:37:00.546795 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:00.546796 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:00.546811 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:37:00.546857 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:00.546858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:00.546870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:00.546924 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:00.547031 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.547430 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.548059 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:00.548227 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:00.548280 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:00.548400 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:00.548475 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:00.548527 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:00.548558 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 18:37:00.548571 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:37:00.550053 21365 raft_consensus.cc:1261] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Refusing update from remote peer 26604490b4ba43f4a693ea98bb2a1390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:00.550074 21231 raft_consensus.cc:1261] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Refusing update from remote peer 26604490b4ba43f4a693ea98bb2a1390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:00.550354 21565 consensus_queue.cc:1035] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:00.550473 21569 consensus_queue.cc:1035] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:00.552428 21572 mvcc.cc:205] Tried to move back new op lower bound from 7334028167370924032 to 7334028167331999744. Current Snapshot: MvccSnapshot[applied={T|T < 7334028167370924032}]
I20260927 18:37:00.552928 21574 mvcc.cc:205] Tried to move back new op lower bound from 7334028167370924032 to 7334028167331999744. Current Snapshot: MvccSnapshot[applied={T|T < 7334028167370924032 or (T in {7334028167370924032})}]
I20260927 18:37:00.553047 21573 mvcc.cc:205] Tried to move back new op lower bound from 7334028167370924032 to 7334028167331999744. Current Snapshot: MvccSnapshot[applied={T|T < 7334028167370924032 or (T in {7334028167370924032})}]
I20260927 18:37:00.554836 21085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:00.554926 21085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:00.555589 21085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Sending DeleteTablet for 3 replicas of tablet ac4475590c9a47a39d5e939e4773a50d
I20260927 18:37:00.555852 21345 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4475590c9a47a39d5e939e4773a50d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:00 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:52624
I20260927 18:37:00.555902 21479 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4475590c9a47a39d5e939e4773a50d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:00 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:52614
I20260927 18:37:00.555917 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21148
I20260927 18:37:00.556589 21583 tablet_replica.cc:330] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: stopping tablet replica
I20260927 18:37:00.556671 21583 raft_consensus.cc:2229] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:00.556761 21583 raft_consensus.cc:2258] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:00.557138 21583 ts_tablet_manager.cc:1905] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:00.558154 21582 tablet_replica.cc:330] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: stopping tablet replica
I20260927 18:37:00.558216 21582 raft_consensus.cc:2229] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:00.558271 21582 raft_consensus.cc:2258] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:00.558457 21583 ts_tablet_manager.cc:1918] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:00.558540 21583 log.cc:1196] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/ac4475590c9a47a39d5e939e4773a50d
I20260927 18:37:00.558808 21583 ts_tablet_manager.cc:1939] T ac4475590c9a47a39d5e939e4773a50d P 26604490b4ba43f4a693ea98bb2a1390: Deleting consensus metadata
I20260927 18:37:00.559185 21071 catalog_manager.cc:4841] TS 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651): tablet ac4475590c9a47a39d5e939e4773a50d (table test-table [id=f9827d49707e493f9e6b56c223a1b996]) successfully deleted
I20260927 18:37:00.559355 21582 ts_tablet_manager.cc:1905] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:00.560329 21582 ts_tablet_manager.cc:1918] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:00.560410 21582 log.cc:1196] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/ac4475590c9a47a39d5e939e4773a50d
I20260927 18:37:00.560631 21582 ts_tablet_manager.cc:1939] T ac4475590c9a47a39d5e939e4773a50d P d6338dc9628a4ae9adcd5a44c4ca6875: Deleting consensus metadata
I20260927 18:37:00.561129 21071 catalog_manager.cc:4841] TS d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109): tablet ac4475590c9a47a39d5e939e4773a50d (table test-table [id=f9827d49707e493f9e6b56c223a1b996]) successfully deleted
W20260927 18:37:00.562363 21072 connection.cc:535] client connection to 127.5.151.193:36705 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 18:37:00.562466 21072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:37:00.562659 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21282
I20260927 18:37:00.563271 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:00.562771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36705 (local address 127.0.0.1:43092)
0927 18:37:00.563202 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:00.563237 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:36705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
W20260927 18:37:00.564744 21072 catalog_manager.cc:4584] TS c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac4475590c9a47a39d5e939e4773a50d: Network error: Client connection negotiation failed: client connection to 127.5.151.193:36705: connect: Connection refused (error 111)
I20260927 18:37:00.567286 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21416
I20260927 18:37:00.571578 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21054
I20260927 18:37:20.576594 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35259
--webserver_interface=127.5.151.254
--webserver_port=42759
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:20.653088 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:20.653259 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:20.653280 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:20.654392 21585 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:37:20.654430 21585 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:37:20.654443 21585 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:37:20.654488 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:20.654513 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:20.654537 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:20.654580 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:20.654599 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:20.656016 21585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:35259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:20.656241 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:20.656453 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:20.664467 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:20.664467 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:20.664469 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:20.664494 21585 server_base.cc:1034] running on GCE node
I20260927 18:37:20.664984 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:20.665241 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:20.666388 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790534240666370 us; error 31 us; skew 500 ppm
Sep 27 18:37:20 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:20.668787 21585 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:37:20.669219 21585 webserver.cc:460] Webserver started at http://127.5.151.254:42759/ using document root <none> and password file <none>
I20260927 18:37:20.669450 21585 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:20.669518 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:20.670791 21585 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:37:20.671376 21601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:20.671540 21585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:20.671624 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "76f6f8f8366347cebd5449f8e071c16a"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-19nl"
server_key: "f35ecc07c0d34fc5337df2bfa54ef181"
server_key_iv: "8b7688546f1ccdbd204197b2672f6943"
server_key_version: "encryptionkey@0"
I20260927 18:37:20.671953 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:20.679937 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:20.681985 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:20.682132 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:20.686244 21585 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:35259
I20260927 18:37:20.686297 21653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:35259 every 8 connection(s)
I20260927 18:37:20.686674 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:20.687929 21654 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:37:20.688530 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Bootstrap starting.
I20260927 18:37:20.691076 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21585
I20260927 18:37:20.691337 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:36705
--local_ip_for_outbound_sockets=127.5.151.193
--tserver_master_addrs=127.5.151.254:35259
--webserver_port=42041
--webserver_interface=127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:20.697126 21654 log.cc:824] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:20.698907 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:37:20.699173 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Bootstrap complete.
I20260927 18:37:20.701311 21654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:20.701570 21654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f6f8f8366347cebd5449f8e071c16a, State: Initialized, Role: FOLLOWER
I20260927 18:37:20.701745 21654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:20.701828 21654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:20.701884 21654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:20.701931 21654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:37:20.702895 21654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:20.703003 21654 leader_election.cc:302] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f6f8f8366347cebd5449f8e071c16a; no voters:
I20260927 18:37:20.703168 21654 leader_election.cc:288] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:37:20.703202 21658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:37:20.703310 21658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 LEADER]: Becoming Leader. State: Replica: 76f6f8f8366347cebd5449f8e071c16a, State: Running, Role: LEADER
I20260927 18:37:20.703426 21654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:20.703449 21658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:20.703703 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76f6f8f8366347cebd5449f8e071c16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } } }
I20260927 18:37:20.703701 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f6f8f8366347cebd5449f8e071c16a. Latest consensus state: current_term: 2 leader_uuid: "76f6f8f8366347cebd5449f8e071c16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } } }
I20260927 18:37:20.703801 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:20.703815 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:20.705183 21674 catalog_manager.cc:1259] Loaded cluster ID: 64cac6bf084046e5be17502c5d402b0d
I20260927 18:37:20.705235 21674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: loading cluster ID for follower catalog manager: success
I20260927 18:37:20.706352 21674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: acquiring CA information for follower catalog manager: success
I20260927 18:37:20.706770 21674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:37:20.706846 21664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:20.707147 21664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:20.707202 21664 catalog_manager.cc:1259] Loaded cluster ID: 64cac6bf084046e5be17502c5d402b0d
I20260927 18:37:20.707252 21664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:20.707396 21664 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:20.707476 21664 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Loaded TSK: 0
I20260927 18:37:20.718082 21664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Generated new TSK 1
I20260927 18:37:20.718159 21664 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:37:20.768257 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:20.768441 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:20.768478 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:20.769580 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:20.769636 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:20.769680 21656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:37:20.771076 21656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:36705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=42041
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:20.771292 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:20.771485 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:37:20.776264 21656 server_base.cc:1034] running on GCE node
W20260927 18:37:20.776242 21682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:20.776255 21681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:20.776458 21684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:20.776683 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:20.776913 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:20.778065 21656 hybrid_clock.cc:648] HybridClock initialized: now 1790534240778053 us; error 30 us; skew 500 ppm
Sep 27 18:37:20 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:20.780581 21656 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:37:20.781070 21656 webserver.cc:460] Webserver started at http://127.5.151.193:42041/ using document root <none> and password file <none>
I20260927 18:37:20.781296 21656 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:20.781378 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:20.782935 21656 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:37:20.783629 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:20.783809 21656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:20.783901 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "c960fca8e06c48d5b08f7a09619846a7"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "60188db7e241fc5d3218df03983f75d1"
server_key_iv: "d91cc18728ed7363599908db1b3ea48a"
server_key_version: "encryptionkey@0"
I20260927 18:37:20.784260 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:20.798307 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:20.800508 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:20.800640 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:20.800855 21656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:20.801258 21698 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:37:20.802481 21656 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:37:20.802538 21656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 18:37:20.802585 21656 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:37:20.803401 21656 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:37:20.803445 21656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 18:37:20.803505 21698 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Bootstrap starting.
I20260927 18:37:20.810405 21656 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:36705
I20260927 18:37:20.810825 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:20.813771 21698 log.cc:824] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:20.814745 21805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:36705 every 8 connection(s)
I20260927 18:37:20.815618 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21656
I20260927 18:37:20.815929 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:33109
--local_ip_for_outbound_sockets=127.5.151.194
--tserver_master_addrs=127.5.151.254:35259
--webserver_port=33155
--webserver_interface=127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:20.817848 21698 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:37:20.818151 21698 tablet_bootstrap.cc:492] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Bootstrap complete.
I20260927 18:37:20.819250 21698 ts_tablet_manager.cc:1397] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.011s
I20260927 18:37:20.821288 21698 raft_consensus.cc:348] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
I20260927 18:37:20.821671 21698 raft_consensus.cc:729] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c960fca8e06c48d5b08f7a09619846a7, State: Initialized, Role: FOLLOWER
I20260927 18:37:20.821825 21698 consensus_queue.cc:260] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } } peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } }
Sep 27 18:37:20 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:20.822090 21698 ts_tablet_manager.cc:1428] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 18:37:20.826749 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:20.811468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:38591)
0927 18:37:20.811785 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:20.811828 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:20.812516 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:20.815427 (+ 2911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:20.815430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:20.815840 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:20.816122 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:20.816128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.817775 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.817777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:20.819051 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:20.819055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.820021 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.820023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:20.820158 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:20.820787 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:20.820797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:20.822607 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:20.824862 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:20.824869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:20.824871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:20.825190 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:20.825485 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:20.825485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:20.825486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:20.825627 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:20.826075 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:20.826082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:20.826244 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 18:37:20.826369 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 18:37:20.826539 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260927 18:37:20.827095 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:20.811856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:38591 (local address 127.5.151.254:35259)
0927 18:37:20.815011 (+ 3155us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:20.815020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:20.815056 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:37:20.815179 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:20.815185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:20.815237 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:20.815389 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:20.816223 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.817646 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.819217 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.819892 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.820473 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:20.822719 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:20.822723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:20.822724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:20.822756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:20.824772 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:20.825286 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:20.825289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:20.825289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:20.825407 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:20.825706 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:20.825707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:20.825707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:20.825996 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:20.826083 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:20.826459 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:37:20.826641 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3107,"thread_start_us":76,"threads_started":1}
I20260927 18:37:20.827663 21806 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35259
I20260927 18:37:20.827749 21806 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:20.827905 21806 heartbeater.cc:511] Master 127.5.151.254:35259 requested a full tablet report, sending...
I20260927 18:37:20.828550 21618 ts_manager.cc:194] Registered new tserver with Master: c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705)
I20260927 18:37:20.829538 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:38591
I20260927 18:37:20.830442 21806 heartbeater.cc:503] Master 127.5.151.254:35259 was elected leader, sending a full tablet report...
I20260927 18:37:20.838816 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:20.829049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36705 (local address 127.0.0.1:50410)
0927 18:37:20.834680 (+ 5631us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:20.834726 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:20.834815 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:20.835023 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:20.835026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:20.835028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:20.835126 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:20.835128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.836345 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.836347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:20.837633 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:20.837636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.838481 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.838482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:20.838602 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:20.838615 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 18:37:20.838684 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:37:20.838746 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5594,"thread_start_us":106,"threads_started":1}
I20260927 18:37:20.838817 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:20.829105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.5.151.193:36705)
0927 18:37:20.830634 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:20.830638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:20.834806 (+ 4168us) server_negotiation.cc:407] Connection header received
0927 18:37:20.834907 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:20.834909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:20.834925 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:20.835008 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:20.835194 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.836243 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.837720 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.838391 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.838464 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:20.838515 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:20.838707 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 18:37:20.838757 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1499,"thread_start_us":84,"threads_started":1}
I20260927 18:37:20.839383 21738 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4475590c9a47a39d5e939e4773a50d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:00 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50410
I20260927 18:37:20.839918 21815 tablet_replica.cc:330] stopping tablet replica
I20260927 18:37:20.840013 21815 raft_consensus.cc:2229] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:20.840067 21815 raft_consensus.cc:2258] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:20.840386 21815 ts_tablet_manager.cc:1905] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:20.842615 21815 ts_tablet_manager.cc:1918] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:20.842684 21815 log.cc:1196] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal/wals/ac4475590c9a47a39d5e939e4773a50d
I20260927 18:37:20.842901 21815 ts_tablet_manager.cc:1939] T ac4475590c9a47a39d5e939e4773a50d P c960fca8e06c48d5b08f7a09619846a7: Deleting consensus metadata
I20260927 18:37:20.843374 21604 catalog_manager.cc:4841] TS c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705): tablet ac4475590c9a47a39d5e939e4773a50d (table test-table [id=f9827d49707e493f9e6b56c223a1b996]) successfully deleted
W20260927 18:37:20.899489 21811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:20.899703 21811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:20.899739 21811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:20.900840 21811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:20.900906 21811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:20.900949 21811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:37:20.902315 21811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:33109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=33155
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:20.902594 21811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:20.902805 21811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:20.907913 21823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:20.907948 21821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:20.908107 21820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:20.908106 21811 server_base.cc:1034] running on GCE node
I20260927 18:37:20.908329 21811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:20.908517 21811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:20.909680 21811 hybrid_clock.cc:648] HybridClock initialized: now 1790534240909665 us; error 27 us; skew 500 ppm
Sep 27 18:37:20 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:20.912038 21811 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:37:20.912424 21811 webserver.cc:460] Webserver started at http://127.5.151.194:33155/ using document root <none> and password file <none>
I20260927 18:37:20.912634 21811 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:20.912709 21811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:20.913975 21811 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:37:20.914476 21830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:20.914634 21811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:20.914713 21811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "d6338dc9628a4ae9adcd5a44c4ca6875"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "3cfc5d9259808647f5b1e2383dfb9480"
server_key_iv: "f8201629c9f7de487124f86a1c1de84e"
server_key_version: "encryptionkey@0"
I20260927 18:37:20.915020 21811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:20.930680 21811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:20.932708 21811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:20.932852 21811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:20.933059 21811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:20.933331 21811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:20.933362 21811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:20.933382 21811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:20.933424 21811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:20.938715 21811 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:33109
I20260927 18:37:20.938784 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:33109 every 8 connection(s)
I20260927 18:37:20.939106 21811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:20.940969 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21811
I20260927 18:37:20.941258 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:37651
--local_ip_for_outbound_sockets=127.5.151.195
--tserver_master_addrs=127.5.151.254:35259
--webserver_port=33285
--webserver_interface=127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:37:20 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:20.948760 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:20.939739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:35223 (local address 127.5.151.254:35259)
0927 18:37:20.939791 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:20.939793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:20.940018 (+ 225us) server_negotiation.cc:407] Connection header received
0927 18:37:20.940453 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:20.940455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:20.940468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:20.940522 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:20.941490 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.942252 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.943230 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.943453 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.943534 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:20.945958 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:20.945959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:20.945960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:20.945966 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:20.947391 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:20.947829 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:20.947830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:20.947830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:20.947894 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:20.948135 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:20.948136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:20.948137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:20.948247 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:20.948314 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:20.948573 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 18:37:20.948668 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:37:20.948896 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:20.939734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:35223)
0927 18:37:20.939911 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:20.939942 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:20.940334 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:20.940590 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:20.940592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:20.941149 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:20.941358 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:20.941365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.942334 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.942335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:20.943146 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:20.943148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:20.943544 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:20.943545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:20.943655 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:20.944077 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:20.944085 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:20.945859 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:20.947488 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:20.947495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:20.947497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:20.947746 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:20.947960 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:20.947961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:20.947961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:20.948064 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:20.948306 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:20.948312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:20.948446 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 18:37:20.948541 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:37:20.948684 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260927 18:37:20.949401 21944 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35259
I20260927 18:37:20.949502 21944 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:20.949661 21944 heartbeater.cc:511] Master 127.5.151.254:35259 requested a full tablet report, sending...
I20260927 18:37:20.950066 21618 ts_manager.cc:194] Registered new tserver with Master: d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109)
I20260927 18:37:20.950619 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:35223
W20260927 18:37:21.019716 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:21.019899 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:21.019922 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:21.020916 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:21.020947 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:21.020972 21947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:37:21.022233 21947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:37651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=33285
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:21.022390 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:21.022586 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:37:21.027027 21947 server_base.cc:1034] running on GCE node
W20260927 18:37:21.027031 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:21.027042 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:21.027130 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:21.027364 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:21.027593 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:21.028741 21947 hybrid_clock.cc:648] HybridClock initialized: now 1790534241028735 us; error 30 us; skew 500 ppm
Sep 27 18:37:21 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534241, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:21.030999 21947 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:37:21.031361 21947 webserver.cc:460] Webserver started at http://127.5.151.195:33285/ using document root <none> and password file <none>
I20260927 18:37:21.031574 21947 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:21.031646 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:21.032795 21947 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:37:21.033309 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:21.033458 21947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:21.033542 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "26604490b4ba43f4a693ea98bb2a1390"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "87ce29dc46c75237d475e56abee5555d"
server_key_iv: "4c99cbe7c2190cfc25670aabd51f7f0e"
server_key_version: "encryptionkey@0"
I20260927 18:37:21.033805 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:21.054306 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:21.055948 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:21.056085 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:21.056280 21947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:21.056556 21947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:21.056602 21947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:21.056638 21947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:21.056665 21947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:21.062086 21947 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:37651
I20260927 18:37:21.062131 22075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:37651 every 8 connection(s)
I20260927 18:37:21.062440 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:21.066092 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 21947
Sep 27 18:37:21 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534241, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:21.070876 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.063119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:47523 (local address 127.5.151.254:35259)
0927 18:37:21.063176 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.063178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.063338 (+ 160us) server_negotiation.cc:407] Connection header received
0927 18:37:21.063707 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.063708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.063717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.063771 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:21.064527 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.064960 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.065806 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.066000 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.066047 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.068198 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:21.068199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:21.068199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:21.068205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:21.069573 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:21.069965 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:21.069966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:21.069967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:21.070024 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:21.070238 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:21.070239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:21.070239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:21.070369 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:21.070458 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.070711 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.070795 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:37:21.071024 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.063103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:47523)
0927 18:37:21.063233 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:21.063268 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:21.063642 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:21.063812 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:21.063815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:21.064300 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:21.064459 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.064463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.065035 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.065036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.065741 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.065743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.066069 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.066071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.066212 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.066671 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:21.066678 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:21.068109 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:21.069638 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:21.069644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:21.069646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:21.069891 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:21.070091 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:21.070091 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:21.070091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:21.070180 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:21.070446 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:21.070450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:21.070578 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 18:37:21.070667 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:37:21.070796 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260927 18:37:21.071425 22076 heartbeater.cc:347] Connected to a master server at 127.5.151.254:35259
I20260927 18:37:21.071506 22076 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:21.071687 22076 heartbeater.cc:511] Master 127.5.151.254:35259 requested a full tablet report, sending...
I20260927 18:37:21.072063 21618 ts_manager.cc:194] Registered new tserver with Master: 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651)
I20260927 18:37:21.072367 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:47523
I20260927 18:37:21.073441 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.066473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.5.151.254:35259)
0927 18:37:21.066583 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.066584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.066700 (+ 116us) server_negotiation.cc:407] Connection header received
0927 18:37:21.066745 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.066746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.066757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.066821 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:21.067465 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.067998 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.068883 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.069183 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.069256 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.070531 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:21.070532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:21.070532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:21.070538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:21.072354 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:21.072746 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:21.072747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:21.072747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:21.072798 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:21.072987 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:21.072988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:21.072989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:21.073082 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:21.073237 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.073270 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.073341 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260927 18:37:21.074191 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:21.076593 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.074492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.5.151.254:35259)
0927 18:37:21.074523 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.074524 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.074624 (+ 100us) server_negotiation.cc:407] Connection header received
0927 18:37:21.074699 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.074700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.074732 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.074798 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:21.074873 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.075324 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.075957 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.076117 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.076192 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.076355 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:21.076409 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 18:37:21.076544 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59116: token expired
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 18:37:21.076633 22080 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59116: token expired
W20260927 18:37:21.076763 21550 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.151.254:35259)
I20260927 18:37:21.082454 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.076994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.5.151.254:35259)
0927 18:37:21.077034 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.077036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.077049 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:37:21.077148 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.077149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.077158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.077207 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:21.077780 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.078450 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.079384 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.079627 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.079677 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.080538 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:21.080539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:21.080539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:21.080544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:21.081643 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:21.081926 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:21.081927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:21.081927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:21.081971 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:21.082079 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:21.082079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:21.082079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:21.082171 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:21.082233 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.082315 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.082366 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:37:21.083339 21618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:21.084698 21618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:21.086768 21738 tablet_service.cc:1467] Processing CreateTablet for tablet ac60464d34ef47e19c798533ada4034a (DEFAULT_TABLE table=test-table [id=3ff31f88950743e5a51b50e0c1f148d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:21.086941 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac60464d34ef47e19c798533ada4034a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:21.088429 21698 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Bootstrap starting.
I20260927 18:37:21.088769 21698 tablet_bootstrap.cc:654] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:21.089289 21698 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: No bootstrap required, opened a new log
I20260927 18:37:21.089339 21698 ts_tablet_manager.cc:1397] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:37:21.089440 21698 raft_consensus.cc:348] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.089499 21698 raft_consensus.cc:374] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:21.089538 21698 raft_consensus.cc:729] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c960fca8e06c48d5b08f7a09619846a7, State: Initialized, Role: FOLLOWER
I20260927 18:37:21.089593 21698 consensus_queue.cc:260] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.089682 21698 ts_tablet_manager.cc:1428] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:21.090210 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.086337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:33109 (local address 127.0.0.1:52736)
0927 18:37:21.086466 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:21.086475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:21.086522 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:21.086790 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:21.086791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:21.086794 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:21.086829 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.086830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.087815 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.087816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.088900 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.088903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.089994 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.089995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.090057 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.090060 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:21.090108 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:37:21.090143 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260927 18:37:21.090430 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.086428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.5.151.194:33109)
0927 18:37:21.086653 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.086657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.086667 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:37:21.086702 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.086703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.086714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.086774 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:21.086885 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.087746 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.088990 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.089839 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.089910 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.089962 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.090311 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.090354 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":112,"threads_started":1}
I20260927 18:37:21.091094 21878 tablet_service.cc:1467] Processing CreateTablet for tablet ac60464d34ef47e19c798533ada4034a (DEFAULT_TABLE table=test-table [id=3ff31f88950743e5a51b50e0c1f148d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:21.091408 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac60464d34ef47e19c798533ada4034a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:21.091624 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.086243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37651 (local address 127.0.0.1:54734)
0927 18:37:21.086288 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:21.086299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:21.086335 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:21.088599 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:21.088600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:21.088602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:21.088635 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.088636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.089735 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.089736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.090733 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.090736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.091388 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.091389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.091457 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.091461 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:37:21.091520 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 18:37:21.091564 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:37:21.091841 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.086478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.5.151.195:37651)
0927 18:37:21.088468 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.088472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.088482 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:37:21.088510 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.088512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.088524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.088598 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:21.088691 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.089627 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.090810 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.091315 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.091380 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.091420 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.091722 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.091773 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1963,"thread_start_us":104,"threads_started":1}
I20260927 18:37:21.092394 22010 tablet_service.cc:1467] Processing CreateTablet for tablet ac60464d34ef47e19c798533ada4034a (DEFAULT_TABLE table=test-table [id=3ff31f88950743e5a51b50e0c1f148d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:21.092676 22010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac60464d34ef47e19c798533ada4034a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:21.094079 22086 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Bootstrap starting.
I20260927 18:37:21.094741 22086 tablet_bootstrap.cc:654] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:21.095291 22086 log.cc:824] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:21.095453 22087 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Bootstrap starting.
I20260927 18:37:21.095937 22086 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: No bootstrap required, opened a new log
I20260927 18:37:21.096024 22087 tablet_bootstrap.cc:654] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:21.096028 22086 ts_tablet_manager.cc:1397] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:37:21.096470 22087 log.cc:824] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:21.097100 22087 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: No bootstrap required, opened a new log
I20260927 18:37:21.097148 22087 ts_tablet_manager.cc:1397] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:37:21.097549 22086 raft_consensus.cc:348] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.097692 22086 raft_consensus.cc:374] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:21.097735 22086 raft_consensus.cc:729] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6338dc9628a4ae9adcd5a44c4ca6875, State: Initialized, Role: FOLLOWER
I20260927 18:37:21.097872 22086 consensus_queue.cc:260] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.098174 22086 ts_tablet_manager.cc:1428] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:37:21.098239 21944 heartbeater.cc:503] Master 127.5.151.254:35259 was elected leader, sending a full tablet report...
I20260927 18:37:21.098778 22087 raft_consensus.cc:348] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.098913 22087 raft_consensus.cc:374] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:21.098954 22087 raft_consensus.cc:729] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26604490b4ba43f4a693ea98bb2a1390, State: Initialized, Role: FOLLOWER
I20260927 18:37:21.099092 22087 consensus_queue.cc:260] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.099313 22087 ts_tablet_manager.cc:1428] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:37:21.099345 22076 heartbeater.cc:503] Master 127.5.151.254:35259 was elected leader, sending a full tablet report...
W20260927 18:37:21.189615 21945 tablet.cc:2369] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:21.312991 22077 tablet.cc:2369] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:21.315233 21807 tablet.cc:2369] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:21.486245 22091 raft_consensus.cc:482] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:21.486377 22091 raft_consensus.cc:504] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.486757 22091 leader_election.cc:288] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109), c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705)
I20260927 18:37:21.489658 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.487010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:35897 (local address 127.5.151.193:36705)
0927 18:37:21.487135 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.487136 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.487145 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:37:21.487202 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.487203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.487213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.487274 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:21.487431 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.488068 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.488996 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.489333 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.489404 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.489414 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.489535 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.489585 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260927 18:37:21.489655 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.486958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36705 (local address 127.5.151.195:35897)
0927 18:37:21.487082 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:21.487091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:21.487143 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:21.487296 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:21.487297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:21.487299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:21.487367 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.487369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.488170 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.488171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.488879 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.488881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.489401 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.489402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.489468 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.489475 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:37:21.489514 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 18:37:21.489555 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260927 18:37:21.489696 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.486975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:59463 (local address 127.5.151.194:33109)
0927 18:37:21.487022 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.487023 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.487094 (+ 71us) server_negotiation.cc:407] Connection header received
0927 18:37:21.487130 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.487131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.487140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.487192 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:21.487357 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.488061 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.488948 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.489339 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.489412 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.489420 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.489583 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.489626 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:37:21.489665 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.486934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:33109 (local address 127.5.151.195:59463)
0927 18:37:21.486997 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:21.487009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:21.487070 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:21.487202 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:21.487202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:21.487204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:21.487277 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.487279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.488128 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.488129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.488866 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:21.488868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.489426 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.489427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:21.489496 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.489512 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:37:21.489564 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 18:37:21.489596 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 18:37:21.489993 21760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac60464d34ef47e19c798533ada4034a" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c960fca8e06c48d5b08f7a09619846a7" is_pre_election: true
I20260927 18:37:21.490167 21760 raft_consensus.cc:2454] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 0.
I20260927 18:37:21.490156 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac60464d34ef47e19c798533ada4034a" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" is_pre_election: true
I20260927 18:37:21.490274 21898 raft_consensus.cc:2454] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 0.
I20260927 18:37:21.490371 21965 leader_election.cc:302] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26604490b4ba43f4a693ea98bb2a1390, c960fca8e06c48d5b08f7a09619846a7; no voters:
I20260927 18:37:21.490495 22091 raft_consensus.cc:2781] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:21.490563 22091 raft_consensus.cc:482] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:21.490595 22091 raft_consensus.cc:3037] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:21.491318 22091 raft_consensus.cc:504] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.491451 22091 leader_election.cc:288] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 election: Requested vote from peers d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109), c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705)
I20260927 18:37:21.491580 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac60464d34ef47e19c798533ada4034a" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875"
I20260927 18:37:21.491616 21760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac60464d34ef47e19c798533ada4034a" candidate_uuid: "26604490b4ba43f4a693ea98bb2a1390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c960fca8e06c48d5b08f7a09619846a7"
I20260927 18:37:21.491658 21898 raft_consensus.cc:3037] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:21.491709 21760 raft_consensus.cc:3037] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:21.492321 21898 raft_consensus.cc:2454] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 1.
I20260927 18:37:21.492385 21760 raft_consensus.cc:2454] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26604490b4ba43f4a693ea98bb2a1390 in term 1.
I20260927 18:37:21.492477 21964 leader_election.cc:302] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875; no voters:
I20260927 18:37:21.492565 22091 raft_consensus.cc:2781] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:21.492694 22091 raft_consensus.cc:686] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 1 LEADER]: Becoming Leader. State: Replica: 26604490b4ba43f4a693ea98bb2a1390, State: Running, Role: LEADER
I20260927 18:37:21.492792 22091 consensus_queue.cc:237] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:21.493573 21618 catalog_manager.cc:5495] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "26604490b4ba43f4a693ea98bb2a1390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20260927 18:37:21.527287 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.525324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.5.151.195:37651)
0927 18:37:21.525376 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:21.525377 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:21.525385 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:37:21.525417 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:21.525418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:21.525435 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:21.525511 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:21.525609 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.526051 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.526703 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:21.526874 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:21.526924 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:21.527030 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:21.527126 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:21.527180 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:21.527199 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 18:37:21.527213 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 18:37:21.528985 21898 raft_consensus.cc:1261] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Refusing update from remote peer 26604490b4ba43f4a693ea98bb2a1390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:21.528985 21760 raft_consensus.cc:1261] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Refusing update from remote peer 26604490b4ba43f4a693ea98bb2a1390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:21.529349 22091 consensus_queue.cc:1035] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:21.529522 22094 consensus_queue.cc:1035] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:21.531575 22096 mvcc.cc:205] Tried to move back new op lower bound from 7334028253299884032 to 7334028253155053568. Current Snapshot: MvccSnapshot[applied={T|T < 7334028253299884032}]
I20260927 18:37:21.531796 22098 mvcc.cc:205] Tried to move back new op lower bound from 7334028253299884032 to 7334028253155053568. Current Snapshot: MvccSnapshot[applied={T|T < 7334028253299884032}]
I20260927 18:37:21.532436 22097 mvcc.cc:205] Tried to move back new op lower bound from 7334028253299884032 to 7334028253155053568. Current Snapshot: MvccSnapshot[applied={T|T < 7334028253299884032}]
I20260927 18:37:21.533990 21618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:21.534085 21618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:21.534797 21618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Sending DeleteTablet for 3 replicas of tablet ac60464d34ef47e19c798533ada4034a
I20260927 18:37:21.535069 22010 tablet_service.cc:1514] Processing DeleteTablet for tablet ac60464d34ef47e19c798533ada4034a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:54734
I20260927 18:37:21.535138 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet ac60464d34ef47e19c798533ada4034a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:52736
I20260927 18:37:21.535208 21738 tablet_service.cc:1514] Processing DeleteTablet for tablet ac60464d34ef47e19c798533ada4034a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:21 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:50410
I20260927 18:37:21.535238 22106 tablet_replica.cc:330] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: stopping tablet replica
I20260927 18:37:21.535326 22106 raft_consensus.cc:2229] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:21.535386 22108 tablet_replica.cc:330] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: stopping tablet replica
I20260927 18:37:21.535435 22108 raft_consensus.cc:2229] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:21.535439 22106 raft_consensus.cc:2258] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:21.535495 22108 raft_consensus.cc:2258] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:21.535513 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21656
I20260927 18:37:21.535843 22106 ts_tablet_manager.cc:1905] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:21.537524 22106 ts_tablet_manager.cc:1918] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:21.537571 22106 log.cc:1196] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/ac60464d34ef47e19c798533ada4034a
I20260927 18:37:21.537746 22106 ts_tablet_manager.cc:1939] T ac60464d34ef47e19c798533ada4034a P 26604490b4ba43f4a693ea98bb2a1390: Deleting consensus metadata
I20260927 18:37:21.542728 21603 catalog_manager.cc:4841] TS 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651): tablet ac60464d34ef47e19c798533ada4034a (table test-table [id=3ff31f88950743e5a51b50e0c1f148d5]) successfully deleted
I20260927 18:37:21.542889 22107 tablet_replica.cc:330] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: stopping tablet replica
W20260927 18:37:21.542891 21604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:36705 (error 108)
I20260927 18:37:21.543006 22107 raft_consensus.cc:2229] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:21.543128 22107 raft_consensus.cc:2258] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:21.543587 22107 ts_tablet_manager.cc:1905] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:37:21.543663 21813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:21.543388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:36705 (local address 127.0.0.1:50416)
0927 18:37:21.543433 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:21.543443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:21.543483 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:21.543628 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:36705: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 18:37:21.543764 21604 catalog_manager.cc:4584] TS c960fca8e06c48d5b08f7a09619846a7 (127.5.151.193:36705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac60464d34ef47e19c798533ada4034a: Network error: Client connection negotiation failed: client connection to 127.5.151.193:36705: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:37:21.544461 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21811
I20260927 18:37:21.548862 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21947
I20260927 18:37:21.548962 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:21.548839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:33109 (local address 127.0.0.1:52748)
0927 18:37:21.548889 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:21.548926 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.194:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:37:21.553603 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 21585
I20260927 18:37:41.558583 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35259
--webserver_interface=127.5.151.254
--webserver_port=42759
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:41.635941 22110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:41.636142 22110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:41.636173 22110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:41.637169 22110 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:37:41.637225 22110 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:37:41.637244 22110 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:37:41.637261 22110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:41.637291 22110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:41.637312 22110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:41.637332 22110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:41.637356 22110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:41.638724 22110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:35259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:41.638970 22110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:41.639185 22110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:37:41.643729 22110 server_base.cc:1034] running on GCE node
W20260927 18:37:41.645987 22115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:41.646077 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:41.645984 22118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:41.646324 22110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:41.646709 22110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:41.647848 22110 hybrid_clock.cc:648] HybridClock initialized: now 1790534261647816 us; error 48 us; skew 500 ppm
Sep 27 18:37:41 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534261, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:41.650319 22110 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:37:41.650840 22110 webserver.cc:460] Webserver started at http://127.5.151.254:42759/ using document root <none> and password file <none>
I20260927 18:37:41.651062 22110 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:41.651104 22110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:41.652453 22110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 18:37:41.653118 22125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:41.653275 22110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:41.653383 22110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "76f6f8f8366347cebd5449f8e071c16a"
format_stamp: "Formatted at 2026-09-27 18:36:59 on dist-test-slave-19nl"
server_key: "f35ecc07c0d34fc5337df2bfa54ef181"
server_key_iv: "8b7688546f1ccdbd204197b2672f6943"
server_key_version: "encryptionkey@0"
I20260927 18:37:41.653697 22110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:41.671761 22110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:41.673570 22110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:41.673713 22110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:41.677364 22110 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:35259
I20260927 18:37:41.677408 22177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:35259 every 8 connection(s)
I20260927 18:37:41.677762 22110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:41.679148 22178 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:37:41.679749 22178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Bootstrap starting.
I20260927 18:37:41.685498 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 22110
I20260927 18:37:41.685755 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:36705
--local_ip_for_outbound_sockets=127.5.151.193
--tserver_master_addrs=127.5.151.254:35259
--webserver_port=42041
--webserver_interface=127.5.151.193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:41.687513 22178 log.cc:824] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:41.689420 22178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:37:41.689682 22178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Bootstrap complete.
I20260927 18:37:41.691953 22178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:41.692174 22178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f6f8f8366347cebd5449f8e071c16a, State: Initialized, Role: FOLLOWER
I20260927 18:37:41.692353 22178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:41.692440 22178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:41.692489 22178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:41.692535 22178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:37:41.693435 22178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:41.693549 22178 leader_election.cc:302] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f6f8f8366347cebd5449f8e071c16a; no voters:
I20260927 18:37:41.693710 22178 leader_election.cc:288] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:37:41.693773 22182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:37:41.693930 22182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [term 3 LEADER]: Becoming Leader. State: Replica: 76f6f8f8366347cebd5449f8e071c16a, State: Running, Role: LEADER
I20260927 18:37:41.693964 22178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:41.694037 22182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } }
I20260927 18:37:41.694324 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f6f8f8366347cebd5449f8e071c16a. Latest consensus state: current_term: 3 leader_uuid: "76f6f8f8366347cebd5449f8e071c16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } } }
I20260927 18:37:41.694404 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:41.694348 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76f6f8f8366347cebd5449f8e071c16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6f8f8366347cebd5449f8e071c16a" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 35259 } } }
I20260927 18:37:41.694615 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:41.695050 22189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:41.695847 22189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:41.695937 22189 catalog_manager.cc:1259] Loaded cluster ID: 64cac6bf084046e5be17502c5d402b0d
I20260927 18:37:41.695962 22189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:41.697046 22189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:41.697196 22189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Loaded TSK: 0
I20260927 18:37:41.697232 22189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76f6f8f8366347cebd5449f8e071c16a: Loaded TSK: 1
F20260927 18:37:41.697435 22189 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f368524e9ad google::LogMessage::Fail() at ??:0
@ 0x7f368525188a google::LogMessage::SendToLog() at ??:0
@ 0x7f368524e4de google::LogMessage::Flush() at ??:0
@ 0x7f368524fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f36879bb104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f36879b8d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f368d1a81cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f368d1a848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f368d1a8594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f368d16ec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f368d178ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f368711c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3687111954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3687f756db start_thread at ??:0
@ 0x7f3683d4c61f clone at ??:0
W20260927 18:37:41.800697 22180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:41.800895 22180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:41.800918 22180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:41.802043 22180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:41.802080 22180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:41.802105 22180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:37:41.803535 22180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:36705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=42041
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:41.803745 22180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:41.803970 22180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:41.808619 22222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:41.808720 22180 server_base.cc:1034] running on GCE node
W20260927 18:37:41.808629 22220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:41.808838 22219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:41.809020 22180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:41.809262 22180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:41.810420 22180 hybrid_clock.cc:648] HybridClock initialized: now 1790534261810411 us; error 27 us; skew 500 ppm
Sep 27 18:37:41 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534261, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:41.812948 22180 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:37:41.813437 22180 webserver.cc:460] Webserver started at http://127.5.151.193:42041/ using document root <none> and password file <none>
I20260927 18:37:41.813664 22180 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:41.813727 22180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:41.815044 22180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:37:41.815683 22229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:41.815852 22180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:41.815928 22180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "c960fca8e06c48d5b08f7a09619846a7"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "60188db7e241fc5d3218df03983f75d1"
server_key_iv: "d91cc18728ed7363599908db1b3ea48a"
server_key_version: "encryptionkey@0"
I20260927 18:37:41.816227 22180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:41.843657 22180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:41.845834 22180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:41.845969 22180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:41.846191 22180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:41.846619 22236 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:37:41.847882 22180 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:37:41.847930 22180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 18:37:41.847954 22180 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:37:41.848529 22180 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:37:41.848568 22180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 18:37:41.848615 22236 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Bootstrap starting.
I20260927 18:37:41.856101 22180 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:36705
I20260927 18:37:41.856271 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:36705 every 8 connection(s)
I20260927 18:37:41.856591 22180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:41.859846 22236 log.cc:824] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:41.860903 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 22180
I20260927 18:37:41.861214 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:33109
--local_ip_for_outbound_sockets=127.5.151.194
--tserver_master_addrs=127.5.151.254:35259
--webserver_port=33155
--webserver_interface=127.5.151.194
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:37:41.861825 22236 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:37:41.862077 22236 tablet_bootstrap.cc:492] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Bootstrap complete.
I20260927 18:37:41.863317 22236 ts_tablet_manager.cc:1397] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260927 18:37:41.865000 22236 raft_consensus.cc:348] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:41.865383 22236 raft_consensus.cc:729] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c960fca8e06c48d5b08f7a09619846a7, State: Initialized, Role: FOLLOWER
I20260927 18:37:41.865549 22236 consensus_queue.cc:260] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:41.865831 22236 ts_tablet_manager.cc:1428] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260927 18:37:41.911029 22346 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:37:41.857272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:58567)
0927 18:37:41.857441 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:41.857479 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:41.858145 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:41.910858 (+ 52713us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
W20260927 18:37:41.911264 22344 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:35259 (0 consecutive failures): Network error: Failed to ping master at 127.5.151.254:35259: Client connection negotiation failed: client connection to 127.5.151.254:35259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:37:41.911613 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:41.911495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:45555)
0927 18:37:41.911549 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:41.911586 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 18:37:41.911984 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:41.911899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:46451)
0927 18:37:41.911950 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:41.911971 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:37:41.912107 22344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:37:41.940445 22348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:41.940611 22348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:41.940634 22348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:41.941779 22348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:41.941817 22348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:41.941869 22348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:37:41.943329 22348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:33109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=33155
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:41.943553 22348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:41.943776 22348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:41.948369 22354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:41.948468 22348 server_base.cc:1034] running on GCE node
W20260927 18:37:41.948381 22357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:41.948426 22355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:41.948791 22348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:41.949146 22348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:41.950330 22348 hybrid_clock.cc:648] HybridClock initialized: now 1790534261950311 us; error 30 us; skew 500 ppm
Sep 27 18:37:41 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534261, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:41.952776 22348 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:37:41.953191 22348 webserver.cc:460] Webserver started at http://127.5.151.194:33155/ using document root <none> and password file <none>
I20260927 18:37:41.953409 22348 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:41.953487 22348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:41.954825 22348 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:37:41.955494 22364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:41.955655 22348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:41.955747 22348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "d6338dc9628a4ae9adcd5a44c4ca6875"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "3cfc5d9259808647f5b1e2383dfb9480"
server_key_iv: "f8201629c9f7de487124f86a1c1de84e"
server_key_version: "encryptionkey@0"
I20260927 18:37:41.956076 22348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:41.978978 22348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:41.981029 22348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:41.981168 22348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:41.981391 22348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:41.981833 22371 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 18:37:41.983105 22371 ts_tablet_manager.cc:1852] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 18:37:41.983357 22371 ts_tablet_manager.cc:1905] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:41.985797 22371 ts_tablet_manager.cc:1918] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:41.985878 22371 log.cc:1196] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/ac60464d34ef47e19c798533ada4034a
I20260927 18:37:41.986145 22371 ts_tablet_manager.cc:1939] T ac60464d34ef47e19c798533ada4034a P d6338dc9628a4ae9adcd5a44c4ca6875: Deleting consensus metadata
I20260927 18:37:41.986392 22348 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 18:37:41.986433 22348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 18:37:41.986464 22348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:41.986490 22348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:41.992158 22348 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:33109
I20260927 18:37:41.992235 22478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:33109 every 8 connection(s)
I20260927 18:37:41.992506 22348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:41.993466 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:41.993142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:43493)
0927 18:37:41.993280 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:41.993339 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260927 18:37:41.993644 22479 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:35259 (0 consecutive failures): Network error: Failed to ping master at 127.5.151.254:35259: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
I20260927 18:37:41.993920 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:41.993831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:52827)
0927 18:37:41.993877 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:41.993900 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:37:41.994187 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:41.994109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:41477)
0927 18:37:41.994150 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:41.994170 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 18:37:41.994277 22479 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:37:41.996490 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 22348
I20260927 18:37:41.996744 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:37651
--local_ip_for_outbound_sockets=127.5.151.195
--tserver_master_addrs=127.5.151.254:35259
--webserver_port=33285
--webserver_interface=127.5.151.195
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:42.070168 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:42.070330 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:42.070353 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:42.071455 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:42.071491 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:42.071514 22482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:37:42.072901 22482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:37651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=33285
--tserver_master_addrs=127.5.151.254:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:42.073067 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:42.073220 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:42.078014 22487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:42.078022 22490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:42.078022 22488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:42.078287 22482 server_base.cc:1034] running on GCE node
I20260927 18:37:42.078502 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:42.078747 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:42.079905 22482 hybrid_clock.cc:648] HybridClock initialized: now 1790534262079896 us; error 25 us; skew 500 ppm
Sep 27 18:37:42 dist-test-slave-19nl krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534262, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:42.082149 22482 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:37:42.082629 22482 webserver.cc:460] Webserver started at http://127.5.151.195:33285/ using document root <none> and password file <none>
I20260927 18:37:42.082834 22482 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:42.082899 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:42.084297 22482 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:37:42.084856 22497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:42.085004 22482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:37:42.085093 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "26604490b4ba43f4a693ea98bb2a1390"
format_stamp: "Formatted at 2026-09-27 18:37:00 on dist-test-slave-19nl"
server_key: "87ce29dc46c75237d475e56abee5555d"
server_key_iv: "4c99cbe7c2190cfc25670aabd51f7f0e"
server_key_version: "encryptionkey@0"
I20260927 18:37:42.085361 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:42.095476 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:42.097409 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:42.097556 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:42.097754 22482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:42.098044 22482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:42.098094 22482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:42.098125 22482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:42.098151 22482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:42.103747 22482 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:37651
I20260927 18:37:42.103789 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:37651 every 8 connection(s)
I20260927 18:37:42.104092 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:42.105074 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:42.104663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:41351)
0927 18:37:42.104802 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:42.104895 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
W20260927 18:37:42.105427 22611 heartbeater.cc:644] Failed to heartbeat to 127.5.151.254:35259 (0 consecutive failures): Network error: Failed to ping master at 127.5.151.254:35259: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
I20260927 18:37:42.105752 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:42.105637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:35369)
0927 18:37:42.105685 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:42.105733 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 18:37:42.106050 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:42.105971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:53973)
0927 18:37:42.106012 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:42.106035 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 18:37:42.106176 22611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:37:42.111052 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 22482
I20260927 18:37:42.912645 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:42.912453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:34111)
0927 18:37:42.912562 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:42.912592 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260927 18:37:42.996457 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:42.996092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:40107)
0927 18:37:42.996344 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:42.996379 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":52,"threads_started":1}
I20260927 18:37:43.106662 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:43.106462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:37855)
0927 18:37:43.106593 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:43.106621 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":45,"threads_started":1}
I20260927 18:37:43.230062 22618 raft_consensus.cc:482] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:43.230171 22618 raft_consensus.cc:504] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:43.230432 22618 leader_election.cc:288] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651), d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109)
Sep 27 18:37:43 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534261, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.195@KRBTEST.COM
Sep 27 18:37:43 dist-test-slave-19nl krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534261, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.194@KRBTEST.COM
I20260927 18:37:43.240440 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:43.230658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:37651 (local address 127.5.151.193:47013)
0927 18:37:43.230716 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:43.230727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:43.230794 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:43.231672 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:43.231676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:43.232083 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:43.232256 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:43.232262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.233298 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.233299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:43.234109 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:43.234111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.234727 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.234728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:43.234819 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:43.235374 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:43.235381 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:43.236901 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:43.238885 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:43.238892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:43.238894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:43.239104 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:43.239381 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:43.239381 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:43.239382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:43.239503 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:43.239932 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:43.239938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:43.240059 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 18:37:43.240162 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:37:43.240335 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":47}
I20260927 18:37:43.240476 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:43.230749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:47013 (local address 127.5.151.195:37651)
0927 18:37:43.230942 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:43.230955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:43.230990 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:37:43.231454 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:43.231456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:43.231497 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:43.231608 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:43.232381 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.233191 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.234200 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.234637 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.234758 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:43.237003 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:43.237006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:43.237007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:43.237018 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:43.238797 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:43.239195 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:43.239198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:43.239199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:43.239304 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:43.239569 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:43.239570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:43.239570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:43.239866 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:43.239926 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:43.240225 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:37:43.240370 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":46,"threads_started":1}
I20260927 18:37:43.240556 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:43.230976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:49997 (local address 127.5.151.194:33109)
0927 18:37:43.231123 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:43.231131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:43.231150 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:37:43.231618 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:43.231621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:43.231669 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:43.231803 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:43.232373 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.233194 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.234195 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.234630 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.234733 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:43.237425 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:43.237428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:43.237429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:43.237439 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:43.239174 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:43.239512 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:43.239515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:43.239515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:43.239617 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:43.239799 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:43.239800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:43.239800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:43.240054 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:43.240136 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:43.240313 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 18:37:43.240439 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260927 18:37:43.240550 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:43.230839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:33109 (local address 127.5.151.193:49997)
0927 18:37:43.230960 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:43.230969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:43.231018 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:43.231795 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:43.231796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:43.232081 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:43.232257 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:43.232262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.233321 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.233322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:43.234107 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:43.234109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:43.234710 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:43.234711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:43.234819 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:43.235418 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:43.235433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:43.237243 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:43.239242 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:43.239243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:43.239243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:43.239446 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:43.239680 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:43.239681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:43.239681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:43.239738 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:43.240123 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:43.240124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:43.240178 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 18:37:43.240248 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:37:43.240408 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":115,"thread_start_us":64,"threads_started":1}
W20260927 18:37:43.241779 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
W20260927 18:37:43.241864 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
I20260927 18:37:43.241916 22231 leader_election.cc:302] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c960fca8e06c48d5b08f7a09619846a7; no voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875
I20260927 18:37:43.242040 22618 raft_consensus.cc:2726] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:37:43.913388 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:43.913146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:43847)
0927 18:37:43.913287 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:43.913332 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":52,"threads_started":1}
I20260927 18:37:43.997031 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:43.996862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:46977)
0927 18:37:43.996969 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:43.996997 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260927 18:37:44.107196 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:44.107026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:52641)
0927 18:37:44.107131 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:44.107163 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260927 18:37:44.837962 22625 raft_consensus.cc:482] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:44.838047 22625 raft_consensus.cc:504] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:44.838165 22625 leader_election.cc:288] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651), d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109)
W20260927 18:37:44.838624 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
W20260927 18:37:44.838694 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
I20260927 18:37:44.838708 22231 leader_election.cc:302] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c960fca8e06c48d5b08f7a09619846a7; no voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875
I20260927 18:37:44.838769 22625 raft_consensus.cc:2726] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:37:44.914179 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:44.913965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:37255)
0927 18:37:44.914094 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:44.914135 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260927 18:37:44.997670 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:44.997485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:46127)
0927 18:37:44.997601 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:44.997634 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260927 18:37:45.107806 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:45.107646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:44213)
0927 18:37:45.107747 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:45.107774 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260927 18:37:45.914845 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:45.914652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:41419)
0927 18:37:45.914764 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:45.914795 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20260927 18:37:45.998322 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:45.998128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:37493)
0927 18:37:45.998257 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:45.998288 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260927 18:37:46.108403 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:46.108212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:43195)
0927 18:37:46.108330 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:46.108366 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260927 18:37:46.915493 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:46.915231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:35137)
0927 18:37:46.915390 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:46.915432 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260927 18:37:46.999055 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:46.998876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:36717)
0927 18:37:46.998989 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:46.999020 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260927 18:37:47.108973 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:47.108785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:34201)
0927 18:37:47.108911 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:47.108943 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260927 18:37:47.916267 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:47.915991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:56965)
0927 18:37:47.916184 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:47.916221 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260927 18:37:47.974905 22636 raft_consensus.cc:482] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:47.974996 22636 raft_consensus.cc:504] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:47.975160 22636 leader_election.cc:288] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651), d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109)
W20260927 18:37:47.975517 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
W20260927 18:37:47.975590 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
I20260927 18:37:47.975606 22231 leader_election.cc:302] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c960fca8e06c48d5b08f7a09619846a7; no voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875
I20260927 18:37:47.975718 22636 raft_consensus.cc:2726] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:37:47.999687 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:47.999487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:42197)
0927 18:37:47.999602 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:47.999633 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260927 18:37:48.109498 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:48.109331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:42747)
0927 18:37:48.109434 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:48.109464 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260927 18:37:48.916965 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:48.916760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:47681)
0927 18:37:48.916885 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:48.916921 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260927 18:37:49.000273 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:49.000085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:41229)
0927 18:37:49.000207 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:49.000238 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260927 18:37:49.110082 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:49.109879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:40207)
0927 18:37:49.110019 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:49.110049 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260927 18:37:49.732210 22642 raft_consensus.cc:482] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:49.732302 22642 raft_consensus.cc:504] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:49.732491 22642 leader_election.cc:288] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651), d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109)
W20260927 18:37:49.732889 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
W20260927 18:37:49.732949 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
I20260927 18:37:49.732963 22231 leader_election.cc:302] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c960fca8e06c48d5b08f7a09619846a7; no voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875
I20260927 18:37:49.733069 22642 raft_consensus.cc:2726] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:37:49.917637 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:49.917442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:39215)
0927 18:37:49.917557 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:49.917591 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 18:37:50.000844 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:50.000653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:39745)
0927 18:37:50.000769 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:50.000808 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260927 18:37:50.110874 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:50.110656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:60425)
0927 18:37:50.110785 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:50.110820 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260927 18:37:50.918231 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:50.918047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:52829)
0927 18:37:50.918154 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:50.918187 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260927 18:37:51.001431 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:51.001232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:60389)
0927 18:37:51.001366 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:51.001396 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260927 18:37:51.111502 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:51.111319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:57101)
0927 18:37:51.111433 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:51.111465 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
I20260927 18:37:51.634636 22649 raft_consensus.cc:482] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:51.634722 22649 raft_consensus.cc:504] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26604490b4ba43f4a693ea98bb2a1390" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 37651 } } peers { permanent_uuid: "d6338dc9628a4ae9adcd5a44c4ca6875" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 33109 } } peers { permanent_uuid: "c960fca8e06c48d5b08f7a09619846a7" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 36705 } }
I20260927 18:37:51.634879 22649 leader_election.cc:288] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651), d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109)
W20260927 18:37:51.635192 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 26604490b4ba43f4a693ea98bb2a1390 (127.5.151.195:37651): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
W20260927 18:37:51.635294 22231 leader_election.cc:341] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6338dc9628a4ae9adcd5a44c4ca6875 (127.5.151.194:33109): Not found: Tablet not found: ac60464d34ef47e19c798533ada4034a
I20260927 18:37:51.635311 22231 leader_election.cc:302] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c960fca8e06c48d5b08f7a09619846a7; no voters: 26604490b4ba43f4a693ea98bb2a1390, d6338dc9628a4ae9adcd5a44c4ca6875
I20260927 18:37:51.635438 22649 raft_consensus.cc:2726] T ac60464d34ef47e19c798533ada4034a P c960fca8e06c48d5b08f7a09619846a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:37:51.918844 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:51.918644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:43321)
0927 18:37:51.918770 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:51.918801 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260927 18:37:52.002014 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:52.001830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:52163)
0927 18:37:52.001949 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:52.001980 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260927 18:37:52.112066 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:52.111890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:51343)
0927 18:37:52.112003 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:52.112032 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 18:37:52.919565 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:52.919349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:36241)
0927 18:37:52.919473 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:52.919508 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260927 18:37:53.002708 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:53.002420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:36379)
0927 18:37:53.002613 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:53.002654 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260927 18:37:53.112627 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:53.112452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:52779)
0927 18:37:53.112558 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:53.112591 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260927 18:37:53.920143 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:53.919926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:43595)
0927 18:37:53.920066 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:53.920099 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260927 18:37:54.003351 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.003138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:48727)
0927 18:37:54.003277 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.003312 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260927 18:37:54.113265 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.113057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:55279)
0927 18:37:54.113191 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.113225 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260927 18:37:54.920753 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:54.920553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:56539)
0927 18:37:54.920678 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:54.920709 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
I20260927 18:37:55.003937 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.003742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:55875)
0927 18:37:55.003867 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.003899 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260927 18:37:55.113816 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.113634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:40039)
0927 18:37:55.113753 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.113784 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260927 18:37:55.921479 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:55.921252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:58719)
0927 18:37:55.921401 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:55.921437 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260927 18:37:56.004573 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.004355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:36685)
0927 18:37:56.004495 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:56.004530 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260927 18:37:56.114462 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.114257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:35823)
0927 18:37:56.114400 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:56.114427 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260927 18:37:56.922179 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:56.921994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.193:59081)
0927 18:37:56.922101 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:56.922131 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260927 18:37:57.005177 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:57.005006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.194:39213)
0927 18:37:57.005110 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:57.005139 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260927 18:37:57.115211 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:57.115008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:35259 (local address 127.5.151.195:44575)
0927 18:37:57.115130 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:57.115166 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260927 18:37:57.118011 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 22180
I20260927 18:37:57.122496 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 22348
I20260927 18:37:57.127194 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 22482
2026-09-27T18:37:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58355 ms)
[----------] 30 tests from SecurityITest (222053 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:57 dist-test-slave-19nl krb5kdc[22670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:57 dist-test-slave-19nl krb5kdc[22670](info): set up 2 sockets
Sep 27 18:37:57 dist-test-slave-19nl krb5kdc[22670](info): commencing operation
krb5kdc: starting...
W20260927 18:37:58.142967 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:37:58.254707 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39503
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:58.334196 22683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:58.334435 22683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:58.334473 22683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:58.335660 22683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:58.335728 22683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:58.335754 22683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:58.335788 22683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:58.335811 22683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:58.337250 22683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:58.337539 22683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:58.337759 22683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:37:58.342783 22683 server_base.cc:1034] running on GCE node
W20260927 18:37:58.342759 22688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.342844 22689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.342767 22691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:58.343246 22683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:58.343484 22683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:58.344652 22683 hybrid_clock.cc:648] HybridClock initialized: now 1790534278344642 us; error 27 us; skew 500 ppm
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:58.347103 22683 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:37:58.347502 22683 webserver.cc:460] Webserver started at http://127.5.151.254:35943/ using document root <none> and password file <none>
I20260927 18:37:58.347748 22683 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:58.347815 22683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:58.347965 22683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:58.349141 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "64c24ca3bf344555a93f2dfad66234fb"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "f9606768d65b1dcaf721f45beadd6759"
server_key_iv: "72c8d51bcc575c922db8c1e629971690"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.349511 22683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "64c24ca3bf344555a93f2dfad66234fb"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "f9606768d65b1dcaf721f45beadd6759"
server_key_iv: "72c8d51bcc575c922db8c1e629971690"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.350853 22683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:37:58.351557 22698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.351724 22683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:58.351822 22683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "64c24ca3bf344555a93f2dfad66234fb"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "f9606768d65b1dcaf721f45beadd6759"
server_key_iv: "72c8d51bcc575c922db8c1e629971690"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.351892 22683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:58.368794 22683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:58.371028 22683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:58.371191 22683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:58.375015 22683 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:39503
I20260927 18:37:58.375051 22750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:39503 every 8 connection(s)
I20260927 18:37:58.375392 22683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:58.376205 22751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:58.378367 22751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb: Bootstrap starting.
I20260927 18:37:58.378726 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 22683
I20260927 18:37:58.378808 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:37:58.379014 5727 external_mini_cluster.cc:1428] Setting key d34a4d42fc7137e0dd0bde71c0f74d73
I20260927 18:37:58.379127 22751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:58.379448 22751 log.cc:824] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:58.380165 22751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb: No bootstrap required, opened a new log
I20260927 18:37:58.381947 22751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c24ca3bf344555a93f2dfad66234fb" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39503 } }
I20260927 18:37:58.382102 22751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:58.382148 22751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c24ca3bf344555a93f2dfad66234fb, State: Initialized, Role: FOLLOWER
I20260927 18:37:58.382274 22751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c24ca3bf344555a93f2dfad66234fb" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39503 } }
I20260927 18:37:58.382368 22751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:58.382411 22751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:58.382457 22751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:58.383029 22751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c24ca3bf344555a93f2dfad66234fb" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39503 } }
I20260927 18:37:58.383147 22751 leader_election.cc:302] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64c24ca3bf344555a93f2dfad66234fb; no voters:
I20260927 18:37:58.383322 22751 leader_election.cc:288] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:58.383360 22756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:58.383482 22756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [term 1 LEADER]: Becoming Leader. State: Replica: 64c24ca3bf344555a93f2dfad66234fb, State: Running, Role: LEADER
I20260927 18:37:58.383584 22751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:58.383620 22756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c24ca3bf344555a93f2dfad66234fb" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39503 } }
I20260927 18:37:58.383945 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c24ca3bf344555a93f2dfad66234fb. Latest consensus state: current_term: 1 leader_uuid: "64c24ca3bf344555a93f2dfad66234fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c24ca3bf344555a93f2dfad66234fb" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39503 } } }
I20260927 18:37:58.384033 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:58.384254 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64c24ca3bf344555a93f2dfad66234fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c24ca3bf344555a93f2dfad66234fb" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39503 } } }
I20260927 18:37:58.384336 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:58.384910 22760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:58.385393 22760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:58.387792 22760 catalog_manager.cc:1347] Generated new cluster ID: 3aae1625dc71429e8a50f585d61f0993
I20260927 18:37:58.387846 22760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:58.389695 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.379532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.5.151.254:39503)
0927 18:37:58.379748 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.379756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.379779 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:37:58.380183 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.380189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.380270 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.380423 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.381002 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.381854 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.382852 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.383349 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.383496 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.386321 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.386324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.386325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.386333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.388127 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.388508 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.388511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.388511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.388603 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.388818 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.388819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.388819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.389041 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.389228 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.389294 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.389422 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260927 18:37:58.396108 22760 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:58.396585 22760 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:58.400983 22760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb: Generated new TSK 0
I20260927 18:37:58.401149 22760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:37:58.430938 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39503
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:58.511051 22777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:58.511258 22777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:58.511281 22777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:58.512420 22777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:58.512463 22777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:58.512490 22777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:37:58.513985 22777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:58.514231 22777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:58.514456 22777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:58.519215 22782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.519212 22785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.519233 22783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:58.519523 22777 server_base.cc:1034] running on GCE node
I20260927 18:37:58.519805 22777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:58.520057 22777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:58.521227 22777 hybrid_clock.cc:648] HybridClock initialized: now 1790534278521214 us; error 29 us; skew 500 ppm
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:58.523890 22777 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:37:58.524341 22777 webserver.cc:460] Webserver started at http://127.5.151.193:43135/ using document root <none> and password file <none>
I20260927 18:37:58.524572 22777 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:58.524653 22777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:58.524803 22777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:58.525826 22777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "d01b860809c84b23b63706c1243422fb"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "b8426717de473652b2424f51fec8bc48"
server_key_iv: "dc87d76c43cfa7fc8c9477911f60f717"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.526185 22777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "d01b860809c84b23b63706c1243422fb"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "b8426717de473652b2424f51fec8bc48"
server_key_iv: "dc87d76c43cfa7fc8c9477911f60f717"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.527567 22777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:37:58.528326 22792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.528509 22777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:37:58.528615 22777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "d01b860809c84b23b63706c1243422fb"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "b8426717de473652b2424f51fec8bc48"
server_key_iv: "dc87d76c43cfa7fc8c9477911f60f717"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.528687 22777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:58.537909 22777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:58.539841 22777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:58.539994 22777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:58.540205 22777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:58.540500 22777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:58.540557 22777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.540606 22777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:58.540652 22777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.546277 22777 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:43433
I20260927 18:37:58.546344 22905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:43433 every 8 connection(s)
I20260927 18:37:58.546736 22777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:58.555208 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 22777
I20260927 18:37:58.555321 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:58.555552 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.547500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:35353 (local address 127.5.151.254:39503)
0927 18:37:58.547542 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.547544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.547758 (+ 214us) server_negotiation.cc:407] Connection header received
0927 18:37:58.548225 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.548226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.548236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.548295 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.549273 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.549788 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.550783 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.550975 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.551027 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.552921 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.552922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.552922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.552928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.554125 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.554539 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.554540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.554541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.554641 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.554855 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.554856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.554856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.554960 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.555013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.555366 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.555456 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:37:58.555553 5727 external_mini_cluster.cc:1428] Setting key 92684d3df46d1c789868657bd4e29662
I20260927 18:37:58.555727 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.547464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39503 (local address 127.5.151.193:35353)
0927 18:37:58.547614 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.547665 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.548111 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.548353 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.548361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.549010 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:58.549194 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.549199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.549864 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.549865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.550714 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.550716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.551043 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.551044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.551135 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.551544 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:58.551552 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:58.552844 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:58.554198 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.554204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.554205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.554474 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.554701 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.554702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.554702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.554797 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.555015 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:58.555033 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:58.555195 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.555319 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.555488 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260927 18:37:58.556174 22906 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39503
I20260927 18:37:58.556252 22906 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:58.556428 22906 heartbeater.cc:511] Master 127.5.151.254:39503 requested a full tablet report, sending...
I20260927 18:37:58.556967 22715 ts_manager.cc:194] Registered new tserver with Master: d01b860809c84b23b63706c1243422fb (127.5.151.193:43433)
I20260927 18:37:58.557857 22715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:35353
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:37:58.577122 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39503
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:58.656854 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:58.657042 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:58.657064 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:58.658182 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:58.658221 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:58.658247 22911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:37:58.659669 22911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:58.659894 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:58.660125 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:58.664938 22916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.665025 22917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.664969 22919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:58.665035 22911 server_base.cc:1034] running on GCE node
I20260927 18:37:58.665375 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:58.665563 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:58.666723 22911 hybrid_clock.cc:648] HybridClock initialized: now 1790534278666702 us; error 36 us; skew 500 ppm
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:58.669034 22911 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:37:58.669449 22911 webserver.cc:460] Webserver started at http://127.5.151.194:42549/ using document root <none> and password file <none>
I20260927 18:37:58.669677 22911 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:58.669745 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:58.669893 22911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:58.670972 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "7d6949e4b4834f6093b0a1a14743dedd"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "26664610c45ad63a368704ceba7f83da"
server_key_iv: "c15b96923aa8fd356f7ee01cd01ed8d4"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.671321 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "7d6949e4b4834f6093b0a1a14743dedd"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "26664610c45ad63a368704ceba7f83da"
server_key_iv: "c15b96923aa8fd356f7ee01cd01ed8d4"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.672574 22911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:37:58.673287 22926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.673563 22911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:37:58.673666 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "7d6949e4b4834f6093b0a1a14743dedd"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "26664610c45ad63a368704ceba7f83da"
server_key_iv: "c15b96923aa8fd356f7ee01cd01ed8d4"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.673738 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:58.689965 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:58.691748 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:58.691903 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:58.692112 22911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:58.692404 22911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:58.692457 22911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.692498 22911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:58.692526 22911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.698307 22911 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:40927
I20260927 18:37:58.698339 23039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:40927 every 8 connection(s)
I20260927 18:37:58.698768 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:58.701396 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 22911
I20260927 18:37:58.701506 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:58.701704 5727 external_mini_cluster.cc:1428] Setting key 0c4c6c3aee70fc101cad2ee49055a9f0
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:58.708459 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.699419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:40265 (local address 127.5.151.254:39503)
0927 18:37:58.699465 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.699467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.699638 (+ 171us) server_negotiation.cc:407] Connection header received
0927 18:37:58.700123 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.700123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.700141 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.700227 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.701060 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.701804 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.703032 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.703352 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.703425 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.705479 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.705481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.705482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.705489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.707166 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.707526 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.707528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.707528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.707589 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.707818 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.707819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.707820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.707968 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.708038 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.708274 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.708355 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:37:58.708483 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.699372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39503 (local address 127.5.151.194:40265)
0927 18:37:58.699519 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.699558 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.700041 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.700283 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.700286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.700815 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:58.700989 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.700993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.701898 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.701899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.702952 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.702954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.703433 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.703434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.703532 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.704063 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:58.704076 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:58.705393 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:58.707244 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.707250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.707252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.707455 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.707665 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.707665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.707666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.707756 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.708039 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:58.708043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:58.708144 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.708226 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.708336 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 18:37:58.708974 23040 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39503
I20260927 18:37:58.709077 23040 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:58.709251 23040 heartbeater.cc:511] Master 127.5.151.254:39503 requested a full tablet report, sending...
I20260927 18:37:58.709626 22715 ts_manager.cc:194] Registered new tserver with Master: 7d6949e4b4834f6093b0a1a14743dedd (127.5.151.194:40927)
I20260927 18:37:58.710143 22715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:40265
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:37:58.726899 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39503
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:37:58.805783 23045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:58.805974 23045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:58.805996 23045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:58.807193 23045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:58.807240 23045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:58.807301 23045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:37:58.809165 23045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:58.809381 23045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:58.809615 23045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:58.814309 23051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.814322 23053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:58.814327 23050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:58.814528 23045 server_base.cc:1034] running on GCE node
I20260927 18:37:58.814739 23045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:58.815013 23045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:58.816185 23045 hybrid_clock.cc:648] HybridClock initialized: now 1790534278816174 us; error 38 us; skew 500 ppm
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:37:58.818452 23045 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:37:58.818881 23045 webserver.cc:460] Webserver started at http://127.5.151.195:40157/ using document root <none> and password file <none>
I20260927 18:37:58.819096 23045 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:58.819167 23045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:58.819319 23045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:58.820297 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "a59b5f9a4ca54f5abe4f1b8417c93878"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "a095b3c254a30e44b4a087ddc01b7dc5"
server_key_iv: "c48b50710695669f14958b253764ce5a"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.820672 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "a59b5f9a4ca54f5abe4f1b8417c93878"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "a095b3c254a30e44b4a087ddc01b7dc5"
server_key_iv: "c48b50710695669f14958b253764ce5a"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.821928 23045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:37:58.822639 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.822815 23045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:37:58.822909 23045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "a59b5f9a4ca54f5abe4f1b8417c93878"
format_stamp: "Formatted at 2026-09-27 18:37:58 on dist-test-slave-19nl"
server_key: "a095b3c254a30e44b4a087ddc01b7dc5"
server_key_iv: "c48b50710695669f14958b253764ce5a"
server_key_version: "encryptionkey@0"
I20260927 18:37:58.822997 23045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:58.837734 23045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:58.839515 23045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:58.839668 23045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:58.839883 23045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:58.840183 23045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:58.840234 23045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.840281 23045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:58.840309 23045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:58.845882 23045 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:33943
I20260927 18:37:58.845917 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:33943 every 8 connection(s)
I20260927 18:37:58.846223 23045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:58.851003 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23045
I20260927 18:37:58.851115 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:58.851316 5727 external_mini_cluster.cc:1428] Setting key 8abf99e87e89246e9e8aadf7ea3157ef
Sep 27 18:37:58 dist-test-slave-19nl krb5kdc[22670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:37:58.855489 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.846947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:42689 (local address 127.5.151.254:39503)
0927 18:37:58.847006 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.847008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.847227 (+ 219us) server_negotiation.cc:407] Connection header received
0927 18:37:58.847743 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.847744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.847757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.847803 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.848720 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.849222 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.850190 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.850390 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.850474 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.852867 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.852868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.852869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.852876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.854076 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.854478 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.854480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.854480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.854541 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.854863 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.854864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.854864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.854974 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.855047 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.855315 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.855391 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:37:58.855706 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.846925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39503 (local address 127.5.151.195:42689)
0927 18:37:58.847104 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.847139 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.847622 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.847882 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.847886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.848438 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:58.848614 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.848619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.849316 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.849317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.850098 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.850100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.850483 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.850484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.850666 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.851201 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:37:58.851211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:58.852778 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:58.854188 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.854194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.854196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.854402 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.854638 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:37:58.854639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:37:58.854639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:37:58.854791 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:37:58.855049 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:58.855055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:58.855189 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.855294 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.855440 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260927 18:37:58.856065 23174 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39503
I20260927 18:37:58.856141 23174 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:58.856336 23174 heartbeater.cc:511] Master 127.5.151.254:39503 requested a full tablet report, sending...
I20260927 18:37:58.856660 22715 ts_manager.cc:194] Registered new tserver with Master: a59b5f9a4ca54f5abe4f1b8417c93878 (127.5.151.195:33943)
I20260927 18:37:58.857021 22715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:42689
I20260927 18:37:58.862936 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:58.870121 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.864162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.5.151.254:39503)
0927 18:37:58.864224 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.864225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.864366 (+ 141us) server_negotiation.cc:407] Connection header received
0927 18:37:58.864463 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.864464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.864474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.864529 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:58.865078 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.865552 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.866291 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.866497 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.866593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.867792 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:58.867793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:58.867794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:37:58.867800 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:58.868978 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.869320 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.869321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.869322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.869374 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:37:58.869573 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:37:58.869574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:37:58.869574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:37:58.869723 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:58.869845 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.869948 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.870027 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:37:58.871484 22715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:58.872964 22715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:58.878943 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.875005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:40927 (local address 127.0.0.1:36480)
0927 18:37:58.875315 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.875354 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.875431 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.875671 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.875674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.875676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:58.875737 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.875739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.876775 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.876776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.877809 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.877812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.878643 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.878644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.878723 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.878737 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.878806 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.878860 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260927 18:37:58.878978 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.875094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.5.151.194:40927)
0927 18:37:58.875345 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.875348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.875412 (+ 64us) server_negotiation.cc:407] Connection header received
0927 18:37:58.875531 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.875533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.875551 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.875643 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:58.875827 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.876677 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.877894 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.878535 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.878624 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.878681 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.878833 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.878879 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260927 18:37:58.879328 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.875027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43433 (local address 127.0.0.1:47330)
0927 18:37:58.875455 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.875464 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.875515 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.875746 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.875747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.875750 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:58.875861 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.875863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.876834 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.876835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.878058 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.878061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.879092 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.879093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.879164 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.879168 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.879218 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.879260 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":62,"threads_started":1}
I20260927 18:37:58.879616 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.875137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.5.151.193:43433)
0927 18:37:58.875342 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.875347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.875511 (+ 164us) server_negotiation.cc:407] Connection header received
0927 18:37:58.875583 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.875585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.875599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.875671 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:58.875943 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.876735 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.878145 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.878886 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.878950 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.879503 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.879525 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.879556 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260927 18:37:58.879725 22974 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0bd3536d0f4c50b42a5df380da4ce1 (DEFAULT_TABLE table=test-table [id=484a931c7da84d198092dbba6e78f380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:58.880057 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0bd3536d0f4c50b42a5df380da4ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:58.880266 22840 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0bd3536d0f4c50b42a5df380da4ce1 (DEFAULT_TABLE table=test-table [id=484a931c7da84d198092dbba6e78f380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:58.880558 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0bd3536d0f4c50b42a5df380da4ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:58.883653 23188 tablet_bootstrap.cc:492] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Bootstrap starting.
I20260927 18:37:58.883764 23189 tablet_bootstrap.cc:492] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb: Bootstrap starting.
I20260927 18:37:58.884433 23189 tablet_bootstrap.cc:654] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:58.884433 23188 tablet_bootstrap.cc:654] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:58.884945 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.875190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:33943 (local address 127.0.0.1:47540)
0927 18:37:58.875599 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:58.875606 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:58.875771 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:58.877049 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:58.877050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:58.877052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:58.877085 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.877087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.878280 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.878281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.879326 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:58.879329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.884705 (+ 5376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.884706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:58.884771 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.884776 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:37:58.884834 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:37:58.884876 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":68,"threads_started":1}
I20260927 18:37:58.884958 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:58.875302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.5.151.195:33943)
0927 18:37:58.875449 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:58.875452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:58.876853 (+ 1401us) server_negotiation.cc:407] Connection header received
0927 18:37:58.876895 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:58.876897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:58.876913 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:58.876985 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:58.877166 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.878176 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.883896 (+ 5720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:58.884596 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:58.884703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:58.884757 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:58.884847 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 18:37:58.884883 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 18:37:58.885068 23189 log.cc:824] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:58.885123 23188 log.cc:824] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:58.885823 23189 tablet_bootstrap.cc:492] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb: No bootstrap required, opened a new log
I20260927 18:37:58.885823 23188 tablet_bootstrap.cc:492] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: No bootstrap required, opened a new log
I20260927 18:37:58.885896 23189 ts_tablet_manager.cc:1397] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:37:58.885896 23188 ts_tablet_manager.cc:1397] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:37:58.887048 23108 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0bd3536d0f4c50b42a5df380da4ce1 (DEFAULT_TABLE table=test-table [id=484a931c7da84d198092dbba6e78f380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:58.887341 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0bd3536d0f4c50b42a5df380da4ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:58.887844 23189 raft_consensus.cc:348] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:58.887844 23188 raft_consensus.cc:348] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:58.888026 23189 raft_consensus.cc:374] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:58.888026 23188 raft_consensus.cc:374] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:58.888064 23189 raft_consensus.cc:729] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01b860809c84b23b63706c1243422fb, State: Initialized, Role: FOLLOWER
I20260927 18:37:58.888064 23188 raft_consensus.cc:729] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d6949e4b4834f6093b0a1a14743dedd, State: Initialized, Role: FOLLOWER
I20260927 18:37:58.888177 23189 consensus_queue.cc:260] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:58.888190 23188 consensus_queue.cc:260] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:58.888448 23189 ts_tablet_manager.cc:1428] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:58.888448 23188 ts_tablet_manager.cc:1428] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:37:58.888567 23040 heartbeater.cc:503] Master 127.5.151.254:39503 was elected leader, sending a full tablet report...
I20260927 18:37:58.888574 22906 heartbeater.cc:503] Master 127.5.151.254:39503 was elected leader, sending a full tablet report...
I20260927 18:37:58.890245 23194 tablet_bootstrap.cc:492] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Bootstrap starting.
I20260927 18:37:58.890931 23194 tablet_bootstrap.cc:654] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:58.891397 23194 log.cc:824] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:58.891974 23194 tablet_bootstrap.cc:492] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: No bootstrap required, opened a new log
I20260927 18:37:58.892025 23194 ts_tablet_manager.cc:1397] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:37:58.893273 23194 raft_consensus.cc:348] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:58.893400 23194 raft_consensus.cc:374] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:58.893421 23194 raft_consensus.cc:729] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a59b5f9a4ca54f5abe4f1b8417c93878, State: Initialized, Role: FOLLOWER
I20260927 18:37:58.893528 23194 consensus_queue.cc:260] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:58.893787 23194 ts_tablet_manager.cc:1428] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:37:58.893823 23174 heartbeater.cc:503] Master 127.5.151.254:39503 was elected leader, sending a full tablet report...
W20260927 18:37:58.949299 23041 tablet.cc:2369] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:37:59.047508 22907 tablet.cc:2369] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:59.078012 23193 raft_consensus.cc:482] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:59.078132 23193 raft_consensus.cc:504] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:59.078462 23193 leader_election.cc:288] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a59b5f9a4ca54f5abe4f1b8417c93878 (127.5.151.195:33943), d01b860809c84b23b63706c1243422fb (127.5.151.193:43433)
I20260927 18:37:59.081336 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.078828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:36905 (local address 127.5.151.195:33943)
0927 18:37:59.078872 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.078874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.078885 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:37:59.078954 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.078954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.078963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.079033 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.079202 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.079668 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.080581 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.080978 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.081063 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.081072 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.081220 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.081255 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:37:59.081344 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.078840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:50445 (local address 127.5.151.193:43433)
0927 18:37:59.078890 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.078892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.078980 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:37:59.079048 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.079050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.079058 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.079149 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.079285 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.079721 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.080595 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.080978 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.081071 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.081079 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.081237 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.081274 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 18:37:59.081357 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.078746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:33943 (local address 127.5.151.194:36905)
0927 18:37:59.078805 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.078820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.078883 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.079038 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.079039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.079040 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.079112 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.079114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.079748 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.079750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.080478 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.080480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.081062 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.081063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.081141 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.081144 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.081216 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.081255 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 18:37:59.081362 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.078746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43433 (local address 127.5.151.194:50445)
0927 18:37:59.078901 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.078911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.078995 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.079141 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.079142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.079143 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.079217 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.079219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.079799 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.079800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.080503 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.080504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.081072 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.081073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.081152 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.081155 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.081228 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.081269 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":13,"thread_start_us":81,"threads_started":1}
I20260927 18:37:59.081686 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0bd3536d0f4c50b42a5df380da4ce1" candidate_uuid: "7d6949e4b4834f6093b0a1a14743dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" is_pre_election: true
I20260927 18:37:59.081670 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0bd3536d0f4c50b42a5df380da4ce1" candidate_uuid: "7d6949e4b4834f6093b0a1a14743dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01b860809c84b23b63706c1243422fb" is_pre_election: true
I20260927 18:37:59.081828 22860 raft_consensus.cc:2454] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d6949e4b4834f6093b0a1a14743dedd in term 0.
I20260927 18:37:59.081828 23128 raft_consensus.cc:2454] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d6949e4b4834f6093b0a1a14743dedd in term 0.
I20260927 18:37:59.082084 22930 leader_election.cc:302] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d6949e4b4834f6093b0a1a14743dedd, a59b5f9a4ca54f5abe4f1b8417c93878; no voters:
I20260927 18:37:59.082247 23193 raft_consensus.cc:2781] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:59.082305 23193 raft_consensus.cc:482] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:59.082334 23193 raft_consensus.cc:3037] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.083033 23193 raft_consensus.cc:504] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:59.083171 23193 leader_election.cc:288] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [CANDIDATE]: Term 1 election: Requested vote from peers a59b5f9a4ca54f5abe4f1b8417c93878 (127.5.151.195:33943), d01b860809c84b23b63706c1243422fb (127.5.151.193:43433)
I20260927 18:37:59.083315 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0bd3536d0f4c50b42a5df380da4ce1" candidate_uuid: "7d6949e4b4834f6093b0a1a14743dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878"
I20260927 18:37:59.083350 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0bd3536d0f4c50b42a5df380da4ce1" candidate_uuid: "7d6949e4b4834f6093b0a1a14743dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01b860809c84b23b63706c1243422fb"
I20260927 18:37:59.083400 23128 raft_consensus.cc:3037] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.083415 22860 raft_consensus.cc:3037] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.084321 22860 raft_consensus.cc:2454] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d6949e4b4834f6093b0a1a14743dedd in term 1.
I20260927 18:37:59.084321 23128 raft_consensus.cc:2454] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d6949e4b4834f6093b0a1a14743dedd in term 1.
I20260927 18:37:59.084527 22930 leader_election.cc:302] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d6949e4b4834f6093b0a1a14743dedd, a59b5f9a4ca54f5abe4f1b8417c93878; no voters:
I20260927 18:37:59.084645 23193 raft_consensus.cc:2781] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:59.084785 23193 raft_consensus.cc:686] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 1 LEADER]: Becoming Leader. State: Replica: 7d6949e4b4834f6093b0a1a14743dedd, State: Running, Role: LEADER
I20260927 18:37:59.084910 23193 consensus_queue.cc:237] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } }
I20260927 18:37:59.085795 22714 catalog_manager.cc:5495] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d6949e4b4834f6093b0a1a14743dedd (127.5.151.194). New cstate: current_term: 1 leader_uuid: "7d6949e4b4834f6093b0a1a14743dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6949e4b4834f6093b0a1a14743dedd" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 40927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 } health_report { overall_health: UNKNOWN } } }
W20260927 18:37:59.096758 23175 tablet.cc:2369] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:37:59.103554 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.101404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.5.151.194:40927)
0927 18:37:59.101446 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.101448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.101455 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:37:59.101533 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.101534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.101548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.101596 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:59.101702 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.102125 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.102872 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.103053 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.103110 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.103269 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:59.103357 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:59.103416 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.103445 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.103461 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:37:59.105265 23128 raft_consensus.cc:1261] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 1 FOLLOWER]: Refusing update from remote peer 7d6949e4b4834f6093b0a1a14743dedd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:59.105265 22860 raft_consensus.cc:1261] T 4e0bd3536d0f4c50b42a5df380da4ce1 P d01b860809c84b23b63706c1243422fb [term 1 FOLLOWER]: Refusing update from remote peer 7d6949e4b4834f6093b0a1a14743dedd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:59.105618 23193 consensus_queue.cc:1035] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d01b860809c84b23b63706c1243422fb" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:59.105754 23198 consensus_queue.cc:1035] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a59b5f9a4ca54f5abe4f1b8417c93878" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:59.108737 23202 mvcc.cc:205] Tried to move back new op lower bound from 7334028407212457984 to 7334028407132356608. Current Snapshot: MvccSnapshot[applied={T|T < 7334028407212457984 or (T in {7334028407212457984})}]
I20260927 18:37:59.110787 22714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:59.110884 22714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:59.111637 22714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64c24ca3bf344555a93f2dfad66234fb: Sending DeleteTablet for 3 replicas of tablet 4e0bd3536d0f4c50b42a5df380da4ce1
I20260927 18:37:59.112002 23108 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0bd3536d0f4c50b42a5df380da4ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:59 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47540
I20260927 18:37:59.111990 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0bd3536d0f4c50b42a5df380da4ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:59 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:47330
I20260927 18:37:59.112246 22974 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0bd3536d0f4c50b42a5df380da4ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:59 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:36480
I20260927 18:37:59.112272 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 22777
I20260927 18:37:59.112291 23211 tablet_replica.cc:330] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: stopping tablet replica
I20260927 18:37:59.112462 23211 raft_consensus.cc:2229] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:59.112696 23211 raft_consensus.cc:2258] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:59.113067 23213 tablet_replica.cc:330] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: stopping tablet replica
I20260927 18:37:59.113173 23213 raft_consensus.cc:2229] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:59.113292 23213 raft_consensus.cc:2258] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:59.113595 23211 ts_tablet_manager.cc:1905] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:59.113721 23213 ts_tablet_manager.cc:1905] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:59.115406 23211 ts_tablet_manager.cc:1918] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:59.115407 23213 ts_tablet_manager.cc:1918] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:59.115476 23213 log.cc:1196] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/4e0bd3536d0f4c50b42a5df380da4ce1
I20260927 18:37:59.115576 23211 log.cc:1196] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/4e0bd3536d0f4c50b42a5df380da4ce1
I20260927 18:37:59.115754 23213 ts_tablet_manager.cc:1939] T 4e0bd3536d0f4c50b42a5df380da4ce1 P 7d6949e4b4834f6093b0a1a14743dedd: Deleting consensus metadata
I20260927 18:37:59.115962 23211 ts_tablet_manager.cc:1939] T 4e0bd3536d0f4c50b42a5df380da4ce1 P a59b5f9a4ca54f5abe4f1b8417c93878: Deleting consensus metadata
I20260927 18:37:59.116201 22702 catalog_manager.cc:4841] TS 7d6949e4b4834f6093b0a1a14743dedd (127.5.151.194:40927): tablet 4e0bd3536d0f4c50b42a5df380da4ce1 (table test-table [id=484a931c7da84d198092dbba6e78f380]) successfully deleted
I20260927 18:37:59.116410 22702 catalog_manager.cc:4841] TS a59b5f9a4ca54f5abe4f1b8417c93878 (127.5.151.195:33943): tablet 4e0bd3536d0f4c50b42a5df380da4ce1 (table test-table [id=484a931c7da84d198092dbba6e78f380]) successfully deleted
W20260927 18:37:59.120608 22701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:43433 (error 108)
I20260927 18:37:59.120795 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 22911
I20260927 18:37:59.121470 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.120922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43433 (local address 127.0.0.1:47334)
0927 18:37:59.121395 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.121430 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454}
W20260927 18:37:59.121723 22701 catalog_manager.cc:4584] TS d01b860809c84b23b63706c1243422fb (127.5.151.193:43433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e0bd3536d0f4c50b42a5df380da4ce1: Network error: Client connection negotiation failed: client connection to 127.5.151.193:43433: connect: Connection refused (error 111)
I20260927 18:37:59.125475 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23045
I20260927 18:37:59.130595 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 22683
2026-09-27T18:37:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2003 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T18:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:37:59Z Disabled control of system clock
I20260927 18:37:59.157618 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39695
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:39695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:59.239209 23218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:59.239408 23218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:59.239427 23218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:59.240429 23218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:37:59.240463 23218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:59.240474 23218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:59.240486 23218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:37:59.240499 23218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:37:59.241798 23218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:39695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:59.241981 23218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:59.242197 23218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:59.246820 23223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.246799 23224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:59.246876 23218 server_base.cc:1034] running on GCE node
W20260927 18:37:59.246786 23226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:59.247207 23218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:59.247522 23218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:59.248680 23218 hybrid_clock.cc:648] HybridClock initialized: now 1790534279248673 us; error 34 us; skew 500 ppm
I20260927 18:37:59.249825 23218 webserver.cc:460] Webserver started at http://127.5.151.254:39395/ using document root <none> and password file <none>
I20260927 18:37:59.250047 23218 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:59.250144 23218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:59.250274 23218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:59.251327 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "c793680f9c614cdcbad5456563bbc404"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "3f8fa23b1f622a1a90cfe8f0a7e04a24"
server_key_iv: "0f03d8600415ac8704d55e6edd62d7ee"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.251667 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "c793680f9c614cdcbad5456563bbc404"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "3f8fa23b1f622a1a90cfe8f0a7e04a24"
server_key_iv: "0f03d8600415ac8704d55e6edd62d7ee"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.252945 23218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:37:59.253647 23232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.253818 23218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:37:59.253912 23218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "c793680f9c614cdcbad5456563bbc404"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "3f8fa23b1f622a1a90cfe8f0a7e04a24"
server_key_iv: "0f03d8600415ac8704d55e6edd62d7ee"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.254005 23218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:59.273275 23218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:59.273604 23218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:59.273759 23218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:59.277633 23218 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:39695
I20260927 18:37:59.277675 23284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:39695 every 8 connection(s)
I20260927 18:37:59.278021 23218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:37:59.278983 23285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:59.281158 23285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404: Bootstrap starting.
I20260927 18:37:59.281541 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23218
I20260927 18:37:59.281674 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:37:59.281869 23285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:59.281889 5727 external_mini_cluster.cc:1428] Setting key 15a5881135480030bae5c2da8dca600e
I20260927 18:37:59.282215 23285 log.cc:824] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:59.283214 23285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404: No bootstrap required, opened a new log
I20260927 18:37:59.285597 23285 raft_consensus.cc:348] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793680f9c614cdcbad5456563bbc404" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39695 } }
I20260927 18:37:59.285729 23285 raft_consensus.cc:374] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:59.285751 23285 raft_consensus.cc:729] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c793680f9c614cdcbad5456563bbc404, State: Initialized, Role: FOLLOWER
I20260927 18:37:59.285887 23285 consensus_queue.cc:260] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793680f9c614cdcbad5456563bbc404" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39695 } }
I20260927 18:37:59.285970 23285 raft_consensus.cc:388] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:37:59.286001 23285 raft_consensus.cc:482] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:37:59.286047 23285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.286759 23285 raft_consensus.cc:504] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793680f9c614cdcbad5456563bbc404" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39695 } }
I20260927 18:37:59.286865 23285 leader_election.cc:302] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c793680f9c614cdcbad5456563bbc404; no voters:
I20260927 18:37:59.287026 23285 leader_election.cc:288] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:37:59.287204 23291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:59.287328 23285 sys_catalog.cc:564] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:37:59.287353 23291 raft_consensus.cc:686] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [term 1 LEADER]: Becoming Leader. State: Replica: c793680f9c614cdcbad5456563bbc404, State: Running, Role: LEADER
I20260927 18:37:59.287479 23291 consensus_queue.cc:237] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793680f9c614cdcbad5456563bbc404" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39695 } }
I20260927 18:37:59.287803 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c793680f9c614cdcbad5456563bbc404. Latest consensus state: current_term: 1 leader_uuid: "c793680f9c614cdcbad5456563bbc404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793680f9c614cdcbad5456563bbc404" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39695 } } }
I20260927 18:37:59.287899 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:59.287819 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c793680f9c614cdcbad5456563bbc404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793680f9c614cdcbad5456563bbc404" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 39695 } } }
I20260927 18:37:59.288173 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404 [sys.catalog]: This master's current role is: LEADER
I20260927 18:37:59.288206 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.282355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.5.151.254:39695)
0927 18:37:59.282580 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.282590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.282632 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:37:59.283155 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.283163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.283236 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.283428 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.283634 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.285075 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.285923 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.286520 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.286719 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.287390 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.287394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.287395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:59.287597 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.287636 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:59.287649 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.287706 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.287914 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.287997 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260927 18:37:59.288296 23299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:37:59.288774 23299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:37:59.290273 23299 catalog_manager.cc:1347] Generated new cluster ID: a0bdb45e62ae4ce18dc1378a701cc7a6
I20260927 18:37:59.290346 23299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:37:59.303226 23299 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:37:59.303740 23299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:37:59.307650 23299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404: Generated new TSK 0
I20260927 18:37:59.307819 23299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:37:59.309674 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39695
--builtin_ntp_servers=127.5.151.212:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:59.406242 23309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:59.406489 23309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:59.406526 23309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:59.407683 23309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:59.407747 23309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:59.407788 23309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:37:59.409072 23309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:59.409350 23309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:59.409636 23309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:59.415230 23315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.415266 23317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.415206 23314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:59.415633 23309 server_base.cc:1034] running on GCE node
I20260927 18:37:59.415849 23309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:59.416124 23309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:59.417309 23309 hybrid_clock.cc:648] HybridClock initialized: now 1790534279417294 us; error 27 us; skew 500 ppm
I20260927 18:37:59.418857 23309 webserver.cc:460] Webserver started at http://127.5.151.193:34543/ using document root <none> and password file <none>
I20260927 18:37:59.419181 23309 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:59.419253 23309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:59.419386 23309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:59.420764 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "5a8753a136004d57989a871f9865f3ae"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "c818dca6853b62f2f1e84e3ad5f0a6fb"
server_key_iv: "40153abb62cb8ba254c6ea0148f89fba"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.421190 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "5a8753a136004d57989a871f9865f3ae"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "c818dca6853b62f2f1e84e3ad5f0a6fb"
server_key_iv: "40153abb62cb8ba254c6ea0148f89fba"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.422905 23309 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260927 18:37:59.423775 23323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.423974 23309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:37:59.424059 23309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "5a8753a136004d57989a871f9865f3ae"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "c818dca6853b62f2f1e84e3ad5f0a6fb"
server_key_iv: "40153abb62cb8ba254c6ea0148f89fba"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.424126 23309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:59.447467 23309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:59.447708 23309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:59.447803 23309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:59.447969 23309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:59.448239 23309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:59.448269 23309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.448292 23309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:59.448303 23309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.454092 23309 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:42711
I20260927 18:37:59.454155 23436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:42711 every 8 connection(s)
I20260927 18:37:59.454460 23309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:37:59.455272 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23309
I20260927 18:37:59.455379 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:37:59.455577 5727 external_mini_cluster.cc:1428] Setting key e232f68caf1148d8dbc26410ffda8cd1
I20260927 18:37:59.456540 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39695
--builtin_ntp_servers=127.5.151.212:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:37:59.460060 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.455296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:57775 (local address 127.5.151.254:39695)
0927 18:37:59.455358 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.455360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.455595 (+ 235us) server_negotiation.cc:407] Connection header received
0927 18:37:59.456133 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.456134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.456145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.456200 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.456979 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.457588 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.458669 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.458853 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.458910 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.459789 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.459790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.459791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:59.459796 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.459811 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:59.459816 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.459869 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.459987 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.460006 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:37:59.460386 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.455285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39695 (local address 127.5.151.193:57775)
0927 18:37:59.455458 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.455497 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.456022 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.456277 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.456281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.456291 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.456882 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.456887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.457680 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.457681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.458581 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.458584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.458939 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.458940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.459096 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.459613 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:59.459613 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.459656 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:59.459684 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.459903 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.459909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.459912 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.459983 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.460089 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260927 18:37:59.460861 23437 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39695
I20260927 18:37:59.460950 23437 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:59.461136 23437 heartbeater.cc:511] Master 127.5.151.254:39695 requested a full tablet report, sending...
I20260927 18:37:59.461618 23249 ts_manager.cc:194] Registered new tserver with Master: 5a8753a136004d57989a871f9865f3ae (127.5.151.193:42711)
I20260927 18:37:59.462220 23249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.193:57775
W20260927 18:37:59.535522 23440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:59.535708 23440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:59.535728 23440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:59.536842 23440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:59.536913 23440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:59.536936 23440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:37:59.538216 23440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:59.538437 23440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:59.538743 23440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:59.543285 23448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.543347 23445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.543272 23446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:59.543586 23440 server_base.cc:1034] running on GCE node
I20260927 18:37:59.543752 23440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:59.543996 23440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:59.545162 23440 hybrid_clock.cc:648] HybridClock initialized: now 1790534279545150 us; error 27 us; skew 500 ppm
I20260927 18:37:59.546538 23440 webserver.cc:460] Webserver started at http://127.5.151.194:41911/ using document root <none> and password file <none>
I20260927 18:37:59.546836 23440 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:59.546891 23440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:59.547003 23440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:59.548082 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "e135d0a494414a50b638c798ef522dda"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "9bcdc39b9f21e80ecf24185acf25091a"
server_key_iv: "836d3c1eef96245e15c147cdb4a702f0"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.548395 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "e135d0a494414a50b638c798ef522dda"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "9bcdc39b9f21e80ecf24185acf25091a"
server_key_iv: "836d3c1eef96245e15c147cdb4a702f0"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.549795 23440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:37:59.550609 23454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.550817 23440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:37:59.550902 23440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "e135d0a494414a50b638c798ef522dda"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "9bcdc39b9f21e80ecf24185acf25091a"
server_key_iv: "836d3c1eef96245e15c147cdb4a702f0"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.550988 23440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:59.574455 23440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:59.574801 23440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:59.574946 23440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:59.575176 23440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:59.575498 23440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:59.575534 23440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.575556 23440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:59.575600 23440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.581079 23440 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:41423
I20260927 18:37:59.581156 23567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:41423 every 8 connection(s)
I20260927 18:37:59.581514 23440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:37:59.586262 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.582272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:36125 (local address 127.5.151.254:39695)
0927 18:37:59.582322 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.582325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.582586 (+ 261us) server_negotiation.cc:407] Connection header received
0927 18:37:59.583088 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.583089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.583098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.583153 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.583735 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.584206 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.585097 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.585278 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.585330 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.586027 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.586028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.586028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:59.586032 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.586045 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:59.586048 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.586088 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.586188 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.586207 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:37:59.586428 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.582265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39695 (local address 127.5.151.194:36125)
0927 18:37:59.582458 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.582490 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.583009 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.583210 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.583213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.583221 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.583670 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.583674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.584282 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.584283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.585022 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.585024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.585350 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.585350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.585447 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.585905 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:59.585906 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.585932 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:59.585954 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.586099 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.586103 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.586105 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.586167 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.586239 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260927 18:37:59.586948 23568 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39695
I20260927 18:37:59.587037 23568 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:59.587167 23568 heartbeater.cc:511] Master 127.5.151.254:39695 requested a full tablet report, sending...
I20260927 18:37:59.587512 23249 ts_manager.cc:194] Registered new tserver with Master: e135d0a494414a50b638c798ef522dda (127.5.151.194:41423)
I20260927 18:37:59.587870 23249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.194:36125
I20260927 18:37:59.591476 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23440
I20260927 18:37:59.591580 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:37:59.591758 5727 external_mini_cluster.cc:1428] Setting key b1e7e9b1b50bc224e50e3270e50f2330
I20260927 18:37:59.592587 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39695
--builtin_ntp_servers=127.5.151.212:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:37:59.670747 23571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:37:59.670933 23571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:37:59.670964 23571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:37:59.672036 23571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:37:59.672096 23571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:37:59.672137 23571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:37:59.673411 23571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:39695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:37:59.673640 23571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:37:59.673895 23571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:37:59.678419 23577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.678428 23576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:37:59.678434 23579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:37:59.678697 23571 server_base.cc:1034] running on GCE node
I20260927 18:37:59.678925 23571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:37:59.679155 23571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:37:59.680308 23571 hybrid_clock.cc:648] HybridClock initialized: now 1790534279680311 us; error 40 us; skew 500 ppm
I20260927 18:37:59.681492 23571 webserver.cc:460] Webserver started at http://127.5.151.195:38161/ using document root <none> and password file <none>
I20260927 18:37:59.681713 23571 fs_manager.cc:359] Metadata directory not provided
I20260927 18:37:59.681780 23571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:37:59.681926 23571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:37:59.682960 23571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "1196db67041c49168138839c29d4ebee"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "cb5e3aba677912c99d8a990aa619f508"
server_key_iv: "c0a799f54477ba4477beab27c392ab29"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.683307 23571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "1196db67041c49168138839c29d4ebee"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "cb5e3aba677912c99d8a990aa619f508"
server_key_iv: "c0a799f54477ba4477beab27c392ab29"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.684500 23571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:37:59.685117 23585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.685276 23571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:37:59.685364 23571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "1196db67041c49168138839c29d4ebee"
format_stamp: "Formatted at 2026-09-27 18:37:59 on dist-test-slave-19nl"
server_key: "cb5e3aba677912c99d8a990aa619f508"
server_key_iv: "c0a799f54477ba4477beab27c392ab29"
server_key_version: "encryptionkey@0"
I20260927 18:37:59.685442 23571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:37:59.712730 23571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:37:59.712975 23571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:37:59.713130 23571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:37:59.713322 23571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:37:59.713603 23571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:37:59.713654 23571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.713683 23571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:37:59.713709 23571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:37:59.719193 23571 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:46065
I20260927 18:37:59.719242 23698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:46065 every 8 connection(s)
I20260927 18:37:59.719538 23571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:37:59.723942 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.720220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:44737 (local address 127.5.151.254:39695)
0927 18:37:59.720264 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.720266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.720458 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:37:59.720852 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.720853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.720863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.720923 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.721464 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.721920 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.722875 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.723051 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.723101 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.723755 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.723756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.723756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:59.723760 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.723772 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:59.723775 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.723821 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.723868 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.723887 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:37:59.724159 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.720189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:39695 (local address 127.5.151.195:44737)
0927 18:37:59.720348 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.720376 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.720774 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.720968 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.720971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.720978 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:37:59.721398 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.721402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.722003 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.722003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.722806 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.722808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.723120 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.723121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.723216 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.723639 (+ 423us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:37:59.723639 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:37:59.723662 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:37:59.723682 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:37:59.723823 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:37:59.723825 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:37:59.723827 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.723871 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.723934 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260927 18:37:59.724682 23699 heartbeater.cc:347] Connected to a master server at 127.5.151.254:39695
I20260927 18:37:59.724778 23699 heartbeater.cc:464] Registering TS with master...
I20260927 18:37:59.724933 23699 heartbeater.cc:511] Master 127.5.151.254:39695 requested a full tablet report, sending...
I20260927 18:37:59.725286 23249 ts_manager.cc:194] Registered new tserver with Master: 1196db67041c49168138839c29d4ebee (127.5.151.195:46065)
I20260927 18:37:59.725592 23249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.195:44737
I20260927 18:37:59.726907 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23571
I20260927 18:37:59.726994 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:37:59.727196 5727 external_mini_cluster.cc:1428] Setting key e17410904d5338e3b7a0b3208c33df22
I20260927 18:37:59.727898 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:37:59.731622 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.729168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.5.151.254:39695)
0927 18:37:59.729234 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.729235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.729346 (+ 111us) server_negotiation.cc:407] Connection header received
0927 18:37:59.729366 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.729367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.729374 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.729421 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:37:59.729550 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.729955 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.730666 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.730834 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.730880 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.731395 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:37:59.731397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:37:59.731397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:37:59.731402 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:37:59.731415 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:37:59.731419 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:37:59.731476 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.731517 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.731545 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:37:59.732831 23249 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:37:59.734356 23249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:37:59.740556 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.736514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42711 (local address 127.0.0.1:39916)
0927 18:37:59.736697 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.736733 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.736792 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.737049 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.737051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.737053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.737127 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.737129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.738509 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.738510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.739486 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.739488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.740258 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.740259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.740332 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.740351 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.740435 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.740486 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":50,"threads_started":1}
I20260927 18:37:59.740566 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.736673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.5.151.193:42711)
0927 18:37:59.736873 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.736877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.736886 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:37:59.736924 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.736925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.736938 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.737037 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.737215 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.738435 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.739580 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.740168 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.740245 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.740330 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.740455 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.740491 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260927 18:37:59.741268 23371 tablet_service.cc:1467] Processing CreateTablet for tablet 79f8b08351654359ade980dd95f1f7e6 (DEFAULT_TABLE table=test-table [id=0635b07920264884a83a0ad3a03afb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:59.741595 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f8b08351654359ade980dd95f1f7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:59.744666 23713 tablet_bootstrap.cc:492] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: Bootstrap starting.
I20260927 18:37:59.744880 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.737077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.5.151.195:46065)
0927 18:37:59.738575 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.738579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.740963 (+ 2384us) server_negotiation.cc:407] Connection header received
0927 18:37:59.741006 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.741008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.741022 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.741103 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.741199 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.742440 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.743664 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.744418 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.744491 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.744571 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.744774 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.744811 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":80,"threads_started":1}
I20260927 18:37:59.745121 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.736911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:46065 (local address 127.0.0.1:48022)
0927 18:37:59.740874 (+ 3963us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.740884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.740945 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.741109 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.741110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.741111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.741138 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.741139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.742525 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.742526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.743571 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.743574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.744605 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.744606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.744685 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.744688 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.744812 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.744857 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3954,"thread_start_us":74,"threads_started":1}
I20260927 18:37:59.745353 23713 tablet_bootstrap.cc:654] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:59.745626 23713 log.cc:824] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:59.745757 23633 tablet_service.cc:1467] Processing CreateTablet for tablet 79f8b08351654359ade980dd95f1f7e6 (DEFAULT_TABLE table=test-table [id=0635b07920264884a83a0ad3a03afb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:59.746071 23633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f8b08351654359ade980dd95f1f7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:59.746364 23713 tablet_bootstrap.cc:492] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: No bootstrap required, opened a new log
I20260927 18:37:59.746440 23713 ts_tablet_manager.cc:1397] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:37:59.748960 23713 raft_consensus.cc:348] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.749109 23713 raft_consensus.cc:374] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:59.749136 23713 raft_consensus.cc:729] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8753a136004d57989a871f9865f3ae, State: Initialized, Role: FOLLOWER
I20260927 18:37:59.749260 23713 consensus_queue.cc:260] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.749356 23715 tablet_bootstrap.cc:492] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Bootstrap starting.
I20260927 18:37:59.749505 23713 ts_tablet_manager.cc:1428] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:37:59.749624 23437 heartbeater.cc:503] Master 127.5.151.254:39695 was elected leader, sending a full tablet report...
I20260927 18:37:59.750100 23715 tablet_bootstrap.cc:654] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:59.750423 23715 log.cc:824] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:59.750607 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.736518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:41423 (local address 127.0.0.1:35892)
0927 18:37:59.737232 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.737241 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.737280 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.746826 (+ 9546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.746827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.746830 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.746883 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.746884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.748177 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.748178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.749237 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.749239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.750371 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.750372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.750430 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.750433 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.750487 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.750522 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":147,"threads_started":1}
I20260927 18:37:59.750618 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.742671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.5.151.194:41423)
0927 18:37:59.746656 (+ 3985us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.746659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.746670 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:37:59.746704 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.746706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.746720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.746959 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.747029 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.748070 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.749510 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.750280 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.750357 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.750435 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.750500 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.750532 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3942,"thread_start_us":85,"threads_started":1}
I20260927 18:37:59.751157 23715 tablet_bootstrap.cc:492] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: No bootstrap required, opened a new log
I20260927 18:37:59.751215 23715 ts_tablet_manager.cc:1397] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 18:37:59.753795 23715 raft_consensus.cc:348] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.753937 23715 raft_consensus.cc:374] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:59.753970 23715 raft_consensus.cc:729] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1196db67041c49168138839c29d4ebee, State: Initialized, Role: FOLLOWER
I20260927 18:37:59.754076 23715 consensus_queue.cc:260] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.754310 23715 ts_tablet_manager.cc:1428] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:37:59.754344 23699 heartbeater.cc:503] Master 127.5.151.254:39695 was elected leader, sending a full tablet report...
I20260927 18:37:59.754782 23502 tablet_service.cc:1467] Processing CreateTablet for tablet 79f8b08351654359ade980dd95f1f7e6 (DEFAULT_TABLE table=test-table [id=0635b07920264884a83a0ad3a03afb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:37:59.755086 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f8b08351654359ade980dd95f1f7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:37:59.758553 23719 tablet_bootstrap.cc:492] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Bootstrap starting.
I20260927 18:37:59.759248 23719 tablet_bootstrap.cc:654] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Neither blocks nor log segments found. Creating new log.
I20260927 18:37:59.759562 23719 log.cc:824] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Log is configured to *not* fsync() on all Append() calls
I20260927 18:37:59.760175 23719 tablet_bootstrap.cc:492] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: No bootstrap required, opened a new log
I20260927 18:37:59.760221 23719 ts_tablet_manager.cc:1397] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:37:59.761838 23719 raft_consensus.cc:348] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.761942 23719 raft_consensus.cc:374] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:37:59.761961 23719 raft_consensus.cc:729] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e135d0a494414a50b638c798ef522dda, State: Initialized, Role: FOLLOWER
I20260927 18:37:59.762063 23719 consensus_queue.cc:260] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.762296 23719 ts_tablet_manager.cc:1428] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:37:59.762365 23568 heartbeater.cc:503] Master 127.5.151.254:39695 was elected leader, sending a full tablet report...
I20260927 18:37:59.773311 23721 raft_consensus.cc:482] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:59.773445 23721 raft_consensus.cc:504] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.773792 23721 leader_election.cc:288] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1196db67041c49168138839c29d4ebee (127.5.151.195:46065), 5a8753a136004d57989a871f9865f3ae (127.5.151.193:42711)
I20260927 18:37:59.774462 23716 raft_consensus.cc:482] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:37:59.774580 23716 raft_consensus.cc:504] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.774852 23716 leader_election.cc:288] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e135d0a494414a50b638c798ef522dda (127.5.151.194:41423), 1196db67041c49168138839c29d4ebee (127.5.151.195:46065)
I20260927 18:37:59.777382 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.773921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42711 (local address 127.5.151.194:35401)
0927 18:37:59.773974 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.773983 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.774020 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.774269 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.774269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.774271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.774329 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.774331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.775299 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.775300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.776421 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.776423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.777177 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.777178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.777239 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.777242 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.777285 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.777320 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:37:59.777643 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.774091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:35401 (local address 127.5.151.193:42711)
0927 18:37:59.774136 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.774138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.774147 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:37:59.774170 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.774171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.774183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.774244 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.774412 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.775123 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.776507 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.777075 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.777129 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.777137 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.777544 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.777579 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:37:59.777861 23391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f8b08351654359ade980dd95f1f7e6" candidate_uuid: "e135d0a494414a50b638c798ef522dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8753a136004d57989a871f9865f3ae" is_pre_election: true
I20260927 18:37:59.777951 23391 raft_consensus.cc:2454] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e135d0a494414a50b638c798ef522dda in term 0.
I20260927 18:37:59.777978 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.774028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:39373 (local address 127.5.151.195:46065)
0927 18:37:59.774069 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.774071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.774307 (+ 236us) server_negotiation.cc:407] Connection header received
0927 18:37:59.774383 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.774384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.774393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.774425 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.775087 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.775766 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.777066 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.777598 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.777637 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.777644 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.777883 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.777917 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:37:59.778131 23455 leader_election.cc:302] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a8753a136004d57989a871f9865f3ae, e135d0a494414a50b638c798ef522dda; no voters:
I20260927 18:37:59.778228 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.773921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:46065 (local address 127.5.151.194:39373)
0927 18:37:59.774253 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.774263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.774317 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.774955 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.774955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.774957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.775013 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.775014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.775856 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.775857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.776977 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.776978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.777724 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.777725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.777802 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.777805 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.778064 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.778101 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":193,"thread_start_us":89,"threads_started":1}
I20260927 18:37:59.778263 23721 raft_consensus.cc:2781] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:37:59.778306 23721 raft_consensus.cc:482] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:37:59.778331 23721 raft_consensus.cc:3037] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.778462 23653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f8b08351654359ade980dd95f1f7e6" candidate_uuid: "e135d0a494414a50b638c798ef522dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1196db67041c49168138839c29d4ebee" is_pre_election: true
I20260927 18:37:59.778594 23653 raft_consensus.cc:2454] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e135d0a494414a50b638c798ef522dda in term 0.
I20260927 18:37:59.778787 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.775107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:46065 (local address 127.5.151.193:60061)
0927 18:37:59.775231 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.775240 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.775282 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.775766 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.775767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.775769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.775822 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.775823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.776684 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.776684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.777795 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.777797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.778516 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.778517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.778601 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.778607 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.778653 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.778698 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260927 18:37:59.779044 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.775508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:60061 (local address 127.5.151.195:46065)
0927 18:37:59.775641 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.775643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.775649 (+ 6us) server_negotiation.cc:407] Connection header received
0927 18:37:59.775667 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.775668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.775675 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.775876 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.775903 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.776574 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.777884 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.778423 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.778879 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.778887 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.778909 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.778951 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260927 18:37:59.779152 23721 raft_consensus.cc:504] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.779212 23653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f8b08351654359ade980dd95f1f7e6" candidate_uuid: "5a8753a136004d57989a871f9865f3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1196db67041c49168138839c29d4ebee" is_pre_election: true
I20260927 18:37:59.779273 23653 raft_consensus.cc:2454] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a8753a136004d57989a871f9865f3ae in term 0.
I20260927 18:37:59.779310 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.774999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:41423 (local address 127.5.151.193:45135)
0927 18:37:59.775040 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.775051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:37:59.775086 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:37:59.775415 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:37:59.775416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:37:59.775417 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:37:59.775496 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.775498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.776451 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.776451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.777651 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:37:59.777654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.779050 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.779051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:37:59.779126 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.779140 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:37:59.779193 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:37:59.779236 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":42}
I20260927 18:37:59.779407 23391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f8b08351654359ade980dd95f1f7e6" candidate_uuid: "e135d0a494414a50b638c798ef522dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8753a136004d57989a871f9865f3ae"
I20260927 18:37:59.779443 23325 leader_election.cc:302] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1196db67041c49168138839c29d4ebee, 5a8753a136004d57989a871f9865f3ae; no voters:
I20260927 18:37:59.779470 23391 raft_consensus.cc:3037] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.779639 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.775282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:45135 (local address 127.5.151.194:41423)
0927 18:37:59.775317 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.775319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.775325 (+ 6us) server_negotiation.cc:407] Connection header received
0927 18:37:59.775345 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.775346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.775354 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.775414 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:37:59.775665 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.776369 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.777993 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.778482 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.778536 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.778542 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.779537 (+ 995us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.779576 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:37:59.779838 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f8b08351654359ade980dd95f1f7e6" candidate_uuid: "5a8753a136004d57989a871f9865f3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135d0a494414a50b638c798ef522dda" is_pre_election: true
I20260927 18:37:59.779953 23522 raft_consensus.cc:2379] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a8753a136004d57989a871f9865f3ae in current term 1: Already voted for candidate e135d0a494414a50b638c798ef522dda in this term.
I20260927 18:37:59.780118 23653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f8b08351654359ade980dd95f1f7e6" candidate_uuid: "e135d0a494414a50b638c798ef522dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1196db67041c49168138839c29d4ebee"
I20260927 18:37:59.780192 23653 raft_consensus.cc:3037] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:37:59.780292 23391 raft_consensus.cc:2454] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e135d0a494414a50b638c798ef522dda in term 1.
I20260927 18:37:59.780400 23721 leader_election.cc:288] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [CANDIDATE]: Term 1 election: Requested vote from peers 1196db67041c49168138839c29d4ebee (127.5.151.195:46065), 5a8753a136004d57989a871f9865f3ae (127.5.151.193:42711)
I20260927 18:37:59.780481 23455 leader_election.cc:302] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a8753a136004d57989a871f9865f3ae, e135d0a494414a50b638c798ef522dda; no voters:
I20260927 18:37:59.780581 23721 raft_consensus.cc:2781] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:37:59.780722 23721 raft_consensus.cc:686] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 1 LEADER]: Becoming Leader. State: Replica: e135d0a494414a50b638c798ef522dda, State: Running, Role: LEADER
I20260927 18:37:59.780803 23721 consensus_queue.cc:237] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } }
I20260927 18:37:59.781076 23653 raft_consensus.cc:2454] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e135d0a494414a50b638c798ef522dda in term 1.
I20260927 18:37:59.781291 23716 raft_consensus.cc:2741] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 18:37:59.781584 23249 catalog_manager.cc:5495] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda reported cstate change: term changed from 0 to 1, leader changed from <none> to e135d0a494414a50b638c798ef522dda (127.5.151.194). New cstate: current_term: 1 leader_uuid: "e135d0a494414a50b638c798ef522dda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e135d0a494414a50b638c798ef522dda" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 41423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 } health_report { overall_health: UNKNOWN } } }
I20260927 18:37:59.786886 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.784618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.5.151.194:41423)
0927 18:37:59.784665 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 18:37:59.784667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:37:59.784676 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:37:59.784749 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:37:59.784750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:37:59.784764 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:37:59.784825 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:37:59.784929 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.785369 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.786113 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:37:59.786305 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:37:59.786376 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:37:59.786540 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:37:59.786658 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:37:59.786740 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:37:59.786788 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:37:59.786805 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:37:59.794947 23391 raft_consensus.cc:1261] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 1 FOLLOWER]: Refusing update from remote peer e135d0a494414a50b638c798ef522dda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:59.795204 23721 consensus_queue.cc:1035] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8753a136004d57989a871f9865f3ae" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:59.795260 23653 raft_consensus.cc:1261] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 1 FOLLOWER]: Refusing update from remote peer e135d0a494414a50b638c798ef522dda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:37:59.795503 23721 consensus_queue.cc:1035] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [LEADER]: Connected to new peer: Peer: permanent_uuid: "1196db67041c49168138839c29d4ebee" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 46065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:37:59.797964 23728 mvcc.cc:205] Tried to move back new op lower bound from 7334028410011881472 to 7334028409982910464. Current Snapshot: MvccSnapshot[applied={T|T < 7334028410011881472}]
I20260927 18:37:59.799422 23729 mvcc.cc:205] Tried to move back new op lower bound from 7334028410011881472 to 7334028409982910464. Current Snapshot: MvccSnapshot[applied={T|T < 7334028410011881472}]
I20260927 18:37:59.800424 23249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:59.800519 23249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:37:59.801244 23249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c793680f9c614cdcbad5456563bbc404: Sending DeleteTablet for 3 replicas of tablet 79f8b08351654359ade980dd95f1f7e6
I20260927 18:37:59.801580 23371 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f8b08351654359ade980dd95f1f7e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:59 UTC) from {username='slave'} at 127.0.0.1:39916
I20260927 18:37:59.801654 23633 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f8b08351654359ade980dd95f1f7e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:59 UTC) from {username='slave'} at 127.0.0.1:48022
I20260927 18:37:59.801807 23738 tablet_replica.cc:330] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: stopping tablet replica
I20260927 18:37:59.801854 23739 tablet_replica.cc:330] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: stopping tablet replica
I20260927 18:37:59.801900 23738 raft_consensus.cc:2229] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:59.801925 23739 raft_consensus.cc:2229] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:37:59.801987 23738 raft_consensus.cc:2258] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:59.801998 23739 raft_consensus.cc:2258] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:59.802084 23502 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f8b08351654359ade980dd95f1f7e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:37:59 UTC) from {username='slave'} at 127.0.0.1:35892
I20260927 18:37:59.802384 23739 ts_tablet_manager.cc:1905] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:59.802455 23738 ts_tablet_manager.cc:1905] T 79f8b08351654359ade980dd95f1f7e6 P 5a8753a136004d57989a871f9865f3ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:59.802645 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23309
I20260927 18:37:59.803228 23740 tablet_replica.cc:330] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: stopping tablet replica
I20260927 18:37:59.803315 23740 raft_consensus.cc:2229] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:37:59.803398 23740 raft_consensus.cc:2258] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:37:59.803829 23740 ts_tablet_manager.cc:1905] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:37:59.804049 23739 ts_tablet_manager.cc:1918] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:59.804107 23739 log.cc:1196] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/79f8b08351654359ade980dd95f1f7e6
I20260927 18:37:59.804345 23739 ts_tablet_manager.cc:1939] T 79f8b08351654359ade980dd95f1f7e6 P 1196db67041c49168138839c29d4ebee: Deleting consensus metadata
I20260927 18:37:59.804785 23234 catalog_manager.cc:4841] TS 1196db67041c49168138839c29d4ebee (127.5.151.195:46065): tablet 79f8b08351654359ade980dd95f1f7e6 (table test-table [id=0635b07920264884a83a0ad3a03afb3f]) successfully deleted
I20260927 18:37:59.805496 23740 ts_tablet_manager.cc:1918] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:37:59.805557 23740 log.cc:1196] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/79f8b08351654359ade980dd95f1f7e6
I20260927 18:37:59.805804 23740 ts_tablet_manager.cc:1939] T 79f8b08351654359ade980dd95f1f7e6 P e135d0a494414a50b638c798ef522dda: Deleting consensus metadata
I20260927 18:37:59.806141 23234 catalog_manager.cc:4841] TS e135d0a494414a50b638c798ef522dda (127.5.151.194:41423): tablet 79f8b08351654359ade980dd95f1f7e6 (table test-table [id=0635b07920264884a83a0ad3a03afb3f]) successfully deleted
W20260927 18:37:59.811189 23233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:42711 (error 108)
I20260927 18:37:59.811674 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:37:59.811534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42711 (local address 127.0.0.1:39922)
0927 18:37:59.811592 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 18:37:59.811637 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 18:37:59.811774 23233 catalog_manager.cc:4584] TS 5a8753a136004d57989a871f9865f3ae (127.5.151.193:42711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79f8b08351654359ade980dd95f1f7e6: Network error: Client connection negotiation failed: client connection to 127.5.151.193:42711: connect: Connection refused (error 111)
I20260927 18:37:59.812289 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23440
I20260927 18:37:59.817642 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23571
I20260927 18:37:59.824410 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23218
2026-09-27T18:37:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (695 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:37:59 dist-test-slave-19nl krb5kdc[23744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:37:59 dist-test-slave-19nl krb5kdc[23744](info): set up 2 sockets
Sep 27 18:37:59 dist-test-slave-19nl krb5kdc[23744](info): commencing operation
krb5kdc: starting...
W20260927 18:38:00.822228 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:00 dist-test-slave-19nl krb5kdc[23744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:00.932932 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:40529
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:01.010185 23757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:01.010416 23757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:01.010450 23757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:01.011539 23757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:01.011605 23757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:01.011636 23757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:01.011667 23757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:01.011689 23757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:01.013087 23757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:40529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:01.013346 23757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:01.013566 23757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:01.018281 23762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:01.018294 23763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:01.018292 23765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:01.018378 23757 server_base.cc:1034] running on GCE node
I20260927 18:38:01.018735 23757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:01.019016 23757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:01.020164 23757 hybrid_clock.cc:648] HybridClock initialized: now 1790534281020145 us; error 35 us; skew 500 ppm
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:01.022516 23757 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:01.022974 23757 webserver.cc:460] Webserver started at http://127.5.151.254:39551/ using document root <none> and password file <none>
I20260927 18:38:01.023212 23757 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:01.023257 23757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:01.023401 23757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:01.024421 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "cb5c0f4284204408bf8e34bed71b0fa5"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "60e07a190aa31fecdc3e615c5c176e43"
server_key_iv: "9b4c042a3f101cd3fe6b267a5d216db7"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.024770 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "cb5c0f4284204408bf8e34bed71b0fa5"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "60e07a190aa31fecdc3e615c5c176e43"
server_key_iv: "9b4c042a3f101cd3fe6b267a5d216db7"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.026054 23757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:01.026850 23772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.027055 23757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:38:01.027160 23757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "cb5c0f4284204408bf8e34bed71b0fa5"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "60e07a190aa31fecdc3e615c5c176e43"
server_key_iv: "9b4c042a3f101cd3fe6b267a5d216db7"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.027245 23757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:01.045061 23757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:01.047267 23757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:01.047416 23757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:01.051137 23757 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:40529
I20260927 18:38:01.051200 23824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:40529 every 8 connection(s)
I20260927 18:38:01.051517 23757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:01.052382 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:01.054628 23825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5: Bootstrap starting.
I20260927 18:38:01.055183 23825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:01.055455 23825 log.cc:824] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:01.056105 23825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5: No bootstrap required, opened a new log
I20260927 18:38:01.057374 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23757
I20260927 18:38:01.057494 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:01.057694 5727 external_mini_cluster.cc:1428] Setting key 4aca5033208935c6f6144b76763d4469
I20260927 18:38:01.057977 23825 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40529 } }
I20260927 18:38:01.058197 23825 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:01.058247 23825 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5c0f4284204408bf8e34bed71b0fa5, State: Initialized, Role: FOLLOWER
I20260927 18:38:01.058377 23825 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40529 } }
I20260927 18:38:01.058444 23825 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:01.058463 23825 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:01.058524 23825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:01.059234 23825 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40529 } }
I20260927 18:38:01.059350 23825 leader_election.cc:302] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5c0f4284204408bf8e34bed71b0fa5; no voters:
I20260927 18:38:01.059513 23825 leader_election.cc:288] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:01.059567 23830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:01.059710 23830 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [term 1 LEADER]: Becoming Leader. State: Replica: cb5c0f4284204408bf8e34bed71b0fa5, State: Running, Role: LEADER
I20260927 18:38:01.059758 23825 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:01.059798 23830 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40529 } }
I20260927 18:38:01.060227 23831 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40529 } } }
I20260927 18:38:01.060310 23831 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:01.060477 23832 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5c0f4284204408bf8e34bed71b0fa5. Latest consensus state: current_term: 1 leader_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5c0f4284204408bf8e34bed71b0fa5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 40529 } } }
I20260927 18:38:01.060532 23832 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5 [sys.catalog]: This master's current role is: LEADER
W20260927 18:38:01.061496 23846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:01.061549 23846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:01.061591 23839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:01.061743 23839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:01.063280 23839 catalog_manager.cc:1347] Generated new cluster ID: 4573e03b080e4b1b96974747e3492272
I20260927 18:38:01.063330 23839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:01.069438 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.058198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.5.151.254:40529)
0927 18:38:01.058393 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.058401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.058457 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:38:01.058599 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.058606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.058660 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.058780 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:01.059622 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.061365 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.062375 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.062955 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.063059 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.065825 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:01.065829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:01.065829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:01.065841 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:01.067885 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.068269 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.068273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.068273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.068386 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.068584 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.068584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.068585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.068879 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:01.069035 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.069115 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.069246 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260927 18:38:01.073132 23839 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:01.073809 23839 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:01.080349 23839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5: Generated new TSK 0
I20260927 18:38:01.080528 23839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:38:01.109160 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40529
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:01.187304 23851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:01.187521 23851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:01.187557 23851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:01.188637 23851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:01.188706 23851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:01.188747 23851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:38:01.190148 23851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:01.190395 23851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:01.190687 23851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:01.195375 23857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:01.195504 23851 server_base.cc:1034] running on GCE node
W20260927 18:38:01.195353 23859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:01.195358 23856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:01.195770 23851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:01.196020 23851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:01.197176 23851 hybrid_clock.cc:648] HybridClock initialized: now 1790534281197161 us; error 26 us; skew 500 ppm
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:01.199658 23851 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:38:01.200079 23851 webserver.cc:460] Webserver started at http://127.5.151.193:45253/ using document root <none> and password file <none>
I20260927 18:38:01.200251 23851 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:01.200289 23851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:01.200373 23851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:01.201352 23851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "de6447989b424b71bc3126e750bf6544"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "828bc4acd3375a74740f9e584367e549"
server_key_iv: "6dd4a9d7f0d508d158125946f93baf8e"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.201632 23851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "de6447989b424b71bc3126e750bf6544"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "828bc4acd3375a74740f9e584367e549"
server_key_iv: "6dd4a9d7f0d508d158125946f93baf8e"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.202840 23851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:01.203509 23866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.203682 23851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:01.203769 23851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "de6447989b424b71bc3126e750bf6544"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "828bc4acd3375a74740f9e584367e549"
server_key_iv: "6dd4a9d7f0d508d158125946f93baf8e"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.203835 23851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:01.214946 23851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:01.216777 23851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:01.216926 23851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:01.217149 23851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:01.217432 23851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:01.217463 23851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.217484 23851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:01.217528 23851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.224135 23851 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:43937
I20260927 18:38:01.224195 23979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:43937 every 8 connection(s)
I20260927 18:38:01.224485 23851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:01.233675 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.225207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:46955 (local address 127.5.151.254:40529)
0927 18:38:01.225250 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.225252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.225479 (+ 227us) server_negotiation.cc:407] Connection header received
0927 18:38:01.225989 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.225990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.226006 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.226070 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:01.227033 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.227545 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.228511 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.228711 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.228795 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.231009 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:01.231010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:01.231011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:01.231019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:01.232311 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.232727 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.232728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.232729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.232789 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.233040 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.233052 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.233053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.233170 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:01.233247 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.233496 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.233577 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:38:01.233862 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23851
I20260927 18:38:01.233886 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.225189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:40529 (local address 127.5.151.193:46955)
0927 18:38:01.225366 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.225399 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.225881 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.226145 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.226148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.226758 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:01.226935 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.226940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.227657 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.227658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.228409 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.228411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.228805 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.228806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.228942 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.229466 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:01.229475 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:01.230912 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:01.232413 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:01.232421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:01.232423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:01.232647 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:01.232866 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:01.232867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:01.232867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:01.232966 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:01.233238 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:01.233244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:01.233369 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.233475 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.233633 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260927 18:38:01.233971 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:01.234174 5727 external_mini_cluster.cc:1428] Setting key a8a1ee86f91d705e5e25b472694dcf63
I20260927 18:38:01.234838 23980 heartbeater.cc:347] Connected to a master server at 127.5.151.254:40529
I20260927 18:38:01.234933 23980 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:01.235098 23980 heartbeater.cc:511] Master 127.5.151.254:40529 requested a full tablet report, sending...
I20260927 18:38:01.235599 23789 ts_manager.cc:194] Registered new tserver with Master: de6447989b424b71bc3126e750bf6544 (127.5.151.193:43937)
I20260927 18:38:01.236176 23789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:46955
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:38:01.255017 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40529
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:01.331851 23985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:01.332027 23985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:01.332048 23985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:01.333143 23985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:01.333209 23985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:01.333258 23985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:38:01.334687 23985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:01.334900 23985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:01.335085 23985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:38:01.339412 23985 server_base.cc:1034] running on GCE node
W20260927 18:38:01.339382 23990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:01.339391 23991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:01.339397 23993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:01.339756 23985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:01.339988 23985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:01.341152 23985 hybrid_clock.cc:648] HybridClock initialized: now 1790534281341135 us; error 37 us; skew 500 ppm
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:01.343372 23985 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:38:01.343780 23985 webserver.cc:460] Webserver started at http://127.5.151.194:42663/ using document root <none> and password file <none>
I20260927 18:38:01.343969 23985 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:01.344028 23985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:01.344167 23985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:01.345033 23985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "caecc5e2288e457c9879edfd02155280"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "f653d6732319eca19073972d90259fcc"
server_key_iv: "264e7f4ebd21572096c15caa81274350"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.345387 23985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "caecc5e2288e457c9879edfd02155280"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "f653d6732319eca19073972d90259fcc"
server_key_iv: "264e7f4ebd21572096c15caa81274350"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.346628 23985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:01.347348 24000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.347518 23985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:01.347608 23985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "caecc5e2288e457c9879edfd02155280"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "f653d6732319eca19073972d90259fcc"
server_key_iv: "264e7f4ebd21572096c15caa81274350"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.347689 23985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:01.355511 23985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:01.357103 23985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:01.357251 23985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:01.357455 23985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:01.357736 23985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:01.357787 23985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.357825 23985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:01.357852 23985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.363802 23985 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:44043
I20260927 18:38:01.363850 24113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:44043 every 8 connection(s)
I20260927 18:38:01.364104 23985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:01.368863 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 23985
I20260927 18:38:01.368968 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:01.369144 5727 external_mini_cluster.cc:1428] Setting key dc79fc590933c68bba59bd07ba0fb5e6
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:01.372804 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.364707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:40529 (local address 127.5.151.194:50153)
0927 18:38:01.364859 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.364888 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.365310 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.365533 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.365537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.366035 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:01.366194 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.366198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.366876 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.366877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.367634 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.367635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.367998 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.367999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.368115 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.368506 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:01.368513 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:01.369908 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:01.371619 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:01.371624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:01.371626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:01.371829 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:01.372036 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:01.372037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:01.372037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:01.372124 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:01.372389 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:01.372393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:01.372501 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.372571 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.372671 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260927 18:38:01.372822 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.364721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:50153 (local address 127.5.151.254:40529)
0927 18:38:01.364775 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.364777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.364957 (+ 180us) server_negotiation.cc:407] Connection header received
0927 18:38:01.365405 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.365406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.365416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.365474 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:01.366282 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.366794 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.367730 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.367911 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.367990 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.369996 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:01.369997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:01.369997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:01.370004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:01.371537 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.371900 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.371901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.371902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.371970 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.372196 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.372197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.372197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.372329 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:01.372396 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.372612 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.372694 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:38:01.373517 24114 heartbeater.cc:347] Connected to a master server at 127.5.151.254:40529
I20260927 18:38:01.373611 24114 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:01.373776 24114 heartbeater.cc:511] Master 127.5.151.254:40529 requested a full tablet report, sending...
I20260927 18:38:01.374133 23789 ts_manager.cc:194] Registered new tserver with Master: caecc5e2288e457c9879edfd02155280 (127.5.151.194:44043)
I20260927 18:38:01.374642 23789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:50153
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:38:01.391144 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40529
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:01.467948 24119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:01.468127 24119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:01.468149 24119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:01.469123 24119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:01.469154 24119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:01.469177 24119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:38:01.470451 24119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:01.470679 24119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:01.470841 24119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:01.475354 24125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:01.475354 24127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:01.475524 24119 server_base.cc:1034] running on GCE node
W20260927 18:38:01.475349 24124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:01.475770 24119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:01.476025 24119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:01.477201 24119 hybrid_clock.cc:648] HybridClock initialized: now 1790534281477179 us; error 31 us; skew 500 ppm
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:01.479629 24119 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:38:01.480036 24119 webserver.cc:460] Webserver started at http://127.5.151.195:44233/ using document root <none> and password file <none>
I20260927 18:38:01.480214 24119 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:01.480253 24119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:01.480332 24119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:01.481208 24119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "a11ee1ce6b8c9b8caf42e4eebb1666f3"
server_key_iv: "e1d0060a21d59b8c39538a722bf1edec"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.481478 24119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "a11ee1ce6b8c9b8caf42e4eebb1666f3"
server_key_iv: "e1d0060a21d59b8c39538a722bf1edec"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.482633 24119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:01.483274 24134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.483479 24119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:01.483549 24119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d"
format_stamp: "Formatted at 2026-09-27 18:38:01 on dist-test-slave-19nl"
server_key: "a11ee1ce6b8c9b8caf42e4eebb1666f3"
server_key_iv: "e1d0060a21d59b8c39538a722bf1edec"
server_key_version: "encryptionkey@0"
I20260927 18:38:01.483595 24119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:01.497256 24119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:01.498873 24119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:01.498971 24119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:01.499110 24119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:01.499316 24119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:01.499344 24119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.499362 24119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:01.499374 24119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:01.504832 24119 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:36533
I20260927 18:38:01.504894 24247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:36533 every 8 connection(s)
I20260927 18:38:01.505164 24119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:38:01 dist-test-slave-19nl krb5kdc[23744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:01.513553 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.505813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:32987 (local address 127.5.151.254:40529)
0927 18:38:01.505853 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.505857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.506036 (+ 179us) server_negotiation.cc:407] Connection header received
0927 18:38:01.506512 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.506512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.506521 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.506676 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:01.507526 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.507992 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.509025 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.509200 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.509247 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.511362 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:01.511364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:01.511364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:01.511370 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:01.512484 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.512821 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.512821 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.512822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.512866 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.513067 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.513067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.513067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.513164 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:01.513211 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.513436 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.513493 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:01.513688 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.505785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:40529 (local address 127.5.151.195:32987)
0927 18:38:01.505935 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.505972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.506428 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.506758 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.506761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.507280 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:01.507451 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.507455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.508142 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.508143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.508947 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.508950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.509271 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.509272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.509380 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.509885 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:01.509893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:01.511271 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:01.512551 (+ 1280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:01.512557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:01.512558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:01.512761 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:01.512921 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:01.512921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:01.512922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:01.513008 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:01.513214 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:01.513218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:01.513317 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.513393 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.513499 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260927 18:38:01.514117 24248 heartbeater.cc:347] Connected to a master server at 127.5.151.254:40529
I20260927 18:38:01.514219 24248 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:01.514379 24248 heartbeater.cc:511] Master 127.5.151.254:40529 requested a full tablet report, sending...
I20260927 18:38:01.514794 23789 ts_manager.cc:194] Registered new tserver with Master: f6a1ed539a3747fbb8eb1cd197bf0d9d (127.5.151.195:36533)
I20260927 18:38:01.515151 23789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:32987
I20260927 18:38:01.515328 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 24119
I20260927 18:38:01.515427 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:01.515605 5727 external_mini_cluster.cc:1428] Setting key 8b34cbe441a6b1a68568cec4913c4cd9
I20260927 18:38:01.516503 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:01.523200 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.517818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.5.151.254:40529)
0927 18:38:01.517854 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.517856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.517961 (+ 105us) server_negotiation.cc:407] Connection header received
0927 18:38:01.518010 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.518011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.518018 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.518058 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:01.518522 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.518973 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.519629 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.519807 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.519851 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.520940 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:01.520941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:01.520942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:01.520947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:01.522243 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.522613 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.522614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.522614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.522661 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:01.522825 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:01.522825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:01.522825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:01.522924 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:01.522990 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.523091 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.523146 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:01.524573 23789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:01.525981 23789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:01.532234 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.528140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:44043 (local address 127.0.0.1:52736)
0927 18:38:01.528224 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.528234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.528283 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.528840 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.528841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.528842 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:01.528882 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.528884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.530015 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.530016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.531030 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.531033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.531950 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.531951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.532022 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.532036 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.532117 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.532181 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":50,"threads_started":1}
I20260927 18:38:01.532259 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.528477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.5.151.194:44043)
0927 18:38:01.528648 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.528652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.528680 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:38:01.528716 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.528718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.528734 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.528836 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:01.528938 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.529908 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.531152 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.531863 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.531928 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.531980 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.532144 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.532182 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260927 18:38:01.532922 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.528016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.5.151.195:36533)
0927 18:38:01.528189 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.528194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.528206 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:38:01.528243 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.528247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.528270 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.528343 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:01.528583 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.529633 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.531234 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.531863 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.531930 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.531979 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.532802 (+ 823us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.532848 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260927 18:38:01.533159 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.527932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:36533 (local address 127.0.0.1:46906)
0927 18:38:01.528035 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.528069 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.528125 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.528423 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.528426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.528428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:01.528499 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.528501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.529735 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.529735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.531153 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.531156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.532631 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.532632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.532710 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.532714 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.533016 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.533081 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":63,"thread_start_us":61,"threads_started":1}
I20260927 18:38:01.533365 24048 tablet_service.cc:1467] Processing CreateTablet for tablet ae0c179c94da42e680d2b32bcb5228b2 (DEFAULT_TABLE table=test-table [id=b15a6698d68d43c497e3675744839ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:01.533772 24048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0c179c94da42e680d2b32bcb5228b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:01.533775 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.528269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43937 (local address 127.0.0.1:34706)
0927 18:38:01.528479 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.528489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.528541 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.529955 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.529956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.529958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:01.530013 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.530014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.531276 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.531277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.531988 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.531990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.533523 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.533525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.533594 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.533599 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.533664 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.533703 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260927 18:38:01.533809 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.528405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.5.151.193:43937)
0927 18:38:01.529801 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.529805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.529813 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:38:01.529847 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.529849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.529882 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.529958 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:01.530098 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.531189 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.532049 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.532882 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.533508 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.533573 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.533682 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.533732 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":81,"threads_started":1}
I20260927 18:38:01.535142 23901 tablet_service.cc:1467] Processing CreateTablet for tablet ae0c179c94da42e680d2b32bcb5228b2 (DEFAULT_TABLE table=test-table [id=b15a6698d68d43c497e3675744839ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:01.535467 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0c179c94da42e680d2b32bcb5228b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:01.535367 24182 tablet_service.cc:1467] Processing CreateTablet for tablet ae0c179c94da42e680d2b32bcb5228b2 (DEFAULT_TABLE table=test-table [id=b15a6698d68d43c497e3675744839ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:01.535678 24182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0c179c94da42e680d2b32bcb5228b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:01.536988 24262 tablet_bootstrap.cc:492] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: Bootstrap starting.
I20260927 18:38:01.537631 24262 tablet_bootstrap.cc:654] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:01.538197 24262 log.cc:824] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:01.538331 24263 tablet_bootstrap.cc:492] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Bootstrap starting.
I20260927 18:38:01.538959 24263 tablet_bootstrap.cc:654] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:01.538949 24264 tablet_bootstrap.cc:492] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: Bootstrap starting.
I20260927 18:38:01.539495 24263 log.cc:824] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:01.539568 24264 tablet_bootstrap.cc:654] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:01.540071 24264 log.cc:824] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:01.540272 24263 tablet_bootstrap.cc:492] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: No bootstrap required, opened a new log
I20260927 18:38:01.540343 24263 ts_tablet_manager.cc:1397] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:38:01.540791 24264 tablet_bootstrap.cc:492] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: No bootstrap required, opened a new log
I20260927 18:38:01.540827 24262 tablet_bootstrap.cc:492] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: No bootstrap required, opened a new log
I20260927 18:38:01.540867 24264 ts_tablet_manager.cc:1397] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:38:01.540881 24262 ts_tablet_manager.cc:1397] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 18:38:01.542194 24264 raft_consensus.cc:348] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.542250 24263 raft_consensus.cc:348] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.542413 24263 raft_consensus.cc:374] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:01.542415 24264 raft_consensus.cc:374] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:01.542470 24264 raft_consensus.cc:729] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6447989b424b71bc3126e750bf6544, State: Initialized, Role: FOLLOWER
I20260927 18:38:01.542471 24263 raft_consensus.cc:729] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a1ed539a3747fbb8eb1cd197bf0d9d, State: Initialized, Role: FOLLOWER
I20260927 18:38:01.542570 24264 consensus_queue.cc:260] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.542641 24263 consensus_queue.cc:260] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.542727 24262 raft_consensus.cc:348] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.542874 24262 raft_consensus.cc:374] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:01.542879 24264 ts_tablet_manager.cc:1428] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:38:01.542901 24263 ts_tablet_manager.cc:1428] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:38:01.542917 24262 raft_consensus.cc:729] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caecc5e2288e457c9879edfd02155280, State: Initialized, Role: FOLLOWER
I20260927 18:38:01.542987 23980 heartbeater.cc:503] Master 127.5.151.254:40529 was elected leader, sending a full tablet report...
I20260927 18:38:01.542997 24248 heartbeater.cc:503] Master 127.5.151.254:40529 was elected leader, sending a full tablet report...
I20260927 18:38:01.543017 24262 consensus_queue.cc:260] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.543272 24262 ts_tablet_manager.cc:1428] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 18:38:01.543324 24114 heartbeater.cc:503] Master 127.5.151.254:40529 was elected leader, sending a full tablet report...
W20260927 18:38:01.614650 24115 tablet.cc:2369] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:01.725200 23981 tablet.cc:2369] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:01.755666 24249 tablet.cc:2369] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:01.813820 24270 raft_consensus.cc:482] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:01.813930 24270 raft_consensus.cc:504] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.814302 24270 leader_election.cc:288] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6a1ed539a3747fbb8eb1cd197bf0d9d (127.5.151.195:36533), de6447989b424b71bc3126e750bf6544 (127.5.151.193:43937)
I20260927 18:38:01.817812 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.814603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:42567 (local address 127.5.151.195:36533)
0927 18:38:01.814659 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.814661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.814671 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:38:01.814698 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.814699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.814708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.814766 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:01.814931 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.815677 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.817074 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.817498 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.817548 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.817556 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.817705 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.817748 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:38:01.817833 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.814469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:36533 (local address 127.5.151.194:42567)
0927 18:38:01.814526 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.814538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.814621 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.814789 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.814790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.814791 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:01.814862 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.814864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.815749 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.815749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.816957 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.816961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.817561 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.817563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.817642 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.817646 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.817695 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.817739 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260927 18:38:01.817873 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.814646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:59861 (local address 127.5.151.193:43937)
0927 18:38:01.814701 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.814702 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.814844 (+ 142us) server_negotiation.cc:407] Connection header received
0927 18:38:01.814886 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.814887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.814913 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.814963 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:01.815102 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.815739 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.817120 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.817512 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.817574 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.817582 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.817766 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.817806 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:38:01.817847 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.814609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43937 (local address 127.5.151.194:59861)
0927 18:38:01.814781 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.814789 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:01.814838 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:01.814973 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:01.814974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:01.814977 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:01.815025 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.815026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.815813 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.815813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.816957 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:01.816961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.817631 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.817632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:01.817702 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.817704 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:38:01.817745 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 18:38:01.817772 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260927 18:38:01.818222 24202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0c179c94da42e680d2b32bcb5228b2" candidate_uuid: "caecc5e2288e457c9879edfd02155280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" is_pre_election: true
I20260927 18:38:01.818256 23934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0c179c94da42e680d2b32bcb5228b2" candidate_uuid: "caecc5e2288e457c9879edfd02155280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de6447989b424b71bc3126e750bf6544" is_pre_election: true
I20260927 18:38:01.818365 24202 raft_consensus.cc:2454] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caecc5e2288e457c9879edfd02155280 in term 0.
I20260927 18:38:01.818365 23934 raft_consensus.cc:2454] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caecc5e2288e457c9879edfd02155280 in term 0.
I20260927 18:38:01.818596 24001 leader_election.cc:302] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caecc5e2288e457c9879edfd02155280, de6447989b424b71bc3126e750bf6544; no voters:
I20260927 18:38:01.818717 24270 raft_consensus.cc:2781] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:01.818799 24270 raft_consensus.cc:482] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:01.818833 24270 raft_consensus.cc:3037] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:01.819487 24270 raft_consensus.cc:504] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.819612 24270 leader_election.cc:288] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [CANDIDATE]: Term 1 election: Requested vote from peers f6a1ed539a3747fbb8eb1cd197bf0d9d (127.5.151.195:36533), de6447989b424b71bc3126e750bf6544 (127.5.151.193:43937)
I20260927 18:38:01.819765 24202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0c179c94da42e680d2b32bcb5228b2" candidate_uuid: "caecc5e2288e457c9879edfd02155280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d"
I20260927 18:38:01.819839 24202 raft_consensus.cc:3037] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:01.820465 24202 raft_consensus.cc:2454] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caecc5e2288e457c9879edfd02155280 in term 1.
I20260927 18:38:01.820588 23934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0c179c94da42e680d2b32bcb5228b2" candidate_uuid: "caecc5e2288e457c9879edfd02155280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de6447989b424b71bc3126e750bf6544"
I20260927 18:38:01.820611 24001 leader_election.cc:302] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caecc5e2288e457c9879edfd02155280, f6a1ed539a3747fbb8eb1cd197bf0d9d; no voters:
I20260927 18:38:01.820669 23934 raft_consensus.cc:3037] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:01.820734 24270 raft_consensus.cc:2781] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:01.820885 24270 raft_consensus.cc:686] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 1 LEADER]: Becoming Leader. State: Replica: caecc5e2288e457c9879edfd02155280, State: Running, Role: LEADER
I20260927 18:38:01.820976 24270 consensus_queue.cc:237] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } }
I20260927 18:38:01.821595 23934 raft_consensus.cc:2454] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caecc5e2288e457c9879edfd02155280 in term 1.
I20260927 18:38:01.821780 23789 catalog_manager.cc:5495] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 reported cstate change: term changed from 0 to 1, leader changed from <none> to caecc5e2288e457c9879edfd02155280 (127.5.151.194). New cstate: current_term: 1 leader_uuid: "caecc5e2288e457c9879edfd02155280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caecc5e2288e457c9879edfd02155280" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:01.885891 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.883609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.5.151.194:44043)
0927 18:38:01.883654 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:01.883656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:01.883665 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:01.883737 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:01.883738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:01.883759 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:01.883829 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:01.883930 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.884374 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.885093 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:01.885329 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:01.885414 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:01.885582 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:01.885674 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:01.885780 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:01.885810 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 18:38:01.885819 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:38:01.887498 23934 raft_consensus.cc:1261] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544 [term 1 FOLLOWER]: Refusing update from remote peer caecc5e2288e457c9879edfd02155280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:01.887513 24202 raft_consensus.cc:1261] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 1 FOLLOWER]: Refusing update from remote peer caecc5e2288e457c9879edfd02155280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:01.887850 24272 consensus_queue.cc:1035] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a1ed539a3747fbb8eb1cd197bf0d9d" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:01.887961 24270 consensus_queue.cc:1035] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de6447989b424b71bc3126e750bf6544" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:01.890268 24277 mvcc.cc:205] Tried to move back new op lower bound from 7334028418608848896 to 7334028418339147776. Current Snapshot: MvccSnapshot[applied={T|T < 7334028418608848896}]
I20260927 18:38:01.892347 23788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:01.892432 23788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:01.893131 23788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb5c0f4284204408bf8e34bed71b0fa5: Sending DeleteTablet for 3 replicas of tablet ae0c179c94da42e680d2b32bcb5228b2
I20260927 18:38:01.893476 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0c179c94da42e680d2b32bcb5228b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:01 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:34706
I20260927 18:38:01.893502 24182 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0c179c94da42e680d2b32bcb5228b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:01 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:46906
I20260927 18:38:01.893667 24285 tablet_replica.cc:330] T ae0c179c94da42e680d2b32bcb5228b2 P de6447989b424b71bc3126e750bf6544: stopping tablet replica
I20260927 18:38:01.893692 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23851
I20260927 18:38:01.893673 24048 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0c179c94da42e680d2b32bcb5228b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:01 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:52736
I20260927 18:38:01.894323 24286 tablet_replica.cc:330] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: stopping tablet replica
I20260927 18:38:01.894408 24286 raft_consensus.cc:2229] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:01.894491 24286 raft_consensus.cc:2258] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:01.894925 24286 ts_tablet_manager.cc:1905] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:01.897018 24286 ts_tablet_manager.cc:1918] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:01.897120 24286 log.cc:1196] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/ae0c179c94da42e680d2b32bcb5228b2
I20260927 18:38:01.897423 24286 ts_tablet_manager.cc:1939] T ae0c179c94da42e680d2b32bcb5228b2 P f6a1ed539a3747fbb8eb1cd197bf0d9d: Deleting consensus metadata
I20260927 18:38:01.897778 23773 catalog_manager.cc:4841] TS f6a1ed539a3747fbb8eb1cd197bf0d9d (127.5.151.195:36533): tablet ae0c179c94da42e680d2b32bcb5228b2 (table test-table [id=b15a6698d68d43c497e3675744839ace]) successfully deleted
W20260927 18:38:01.899742 23773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:43937 (error 108)
I20260927 18:38:01.899919 24287 tablet_replica.cc:330] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280: stopping tablet replica
I20260927 18:38:01.900017 24287 raft_consensus.cc:2229] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:01.900137 24287 raft_consensus.cc:2258] T ae0c179c94da42e680d2b32bcb5228b2 P caecc5e2288e457c9879edfd02155280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:01.900168 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.900058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43937 (local address 127.0.0.1:34712)
0927 18:38:01.900099 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.900128 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:43937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 18:38:01.900259 23773 catalog_manager.cc:4584] TS de6447989b424b71bc3126e750bf6544 (127.5.151.193:43937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae0c179c94da42e680d2b32bcb5228b2: Network error: Client connection negotiation failed: client connection to 127.5.151.193:43937: connect: Connection refused (error 111)
I20260927 18:38:01.900305 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23985
I20260927 18:38:01.905481 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:01.905374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:44043 (local address 127.0.0.1:52764)
0927 18:38:01.905422 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:01.905450 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.194:44043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:38:01.905505 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 24119
I20260927 18:38:01.909463 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 23757
2026-09-27T18:38:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2081 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T18:38:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:01Z Disabled control of system clock
I20260927 18:38:01.935377 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:46505
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:46505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:38:02.013947 24292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:02.014137 24292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:02.014155 24292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:02.015241 24292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:02.015278 24292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:02.015291 24292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:02.015303 24292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:02.015316 24292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:02.016474 24292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:46505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:46505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:02.016634 24292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:02.016830 24292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:02.020987 24300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:02.021039 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:02.020990 24298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:02.021188 24292 server_base.cc:1034] running on GCE node
I20260927 18:38:02.021448 24292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:02.021719 24292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:02.022848 24292 hybrid_clock.cc:648] HybridClock initialized: now 1790534282022840 us; error 29 us; skew 500 ppm
I20260927 18:38:02.023902 24292 webserver.cc:460] Webserver started at http://127.5.151.254:36469/ using document root <none> and password file <none>
I20260927 18:38:02.024118 24292 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:02.024160 24292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:02.024298 24292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:02.025235 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "34a366d102b74f73a83d7951be572ff2"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "da4b8f4eecf6260051f86b769b42b353"
server_key_iv: "cfd4964dade1c501736ae6a6047cd026"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.025563 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "34a366d102b74f73a83d7951be572ff2"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "da4b8f4eecf6260051f86b769b42b353"
server_key_iv: "cfd4964dade1c501736ae6a6047cd026"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.026754 24292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:02.027482 24306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.027652 24292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:02.027745 24292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "34a366d102b74f73a83d7951be572ff2"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "da4b8f4eecf6260051f86b769b42b353"
server_key_iv: "cfd4964dade1c501736ae6a6047cd026"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.027820 24292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:02.040529 24292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:02.040768 24292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:02.040892 24292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:02.044378 24292 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:46505
I20260927 18:38:02.044409 24358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:46505 every 8 connection(s)
I20260927 18:38:02.044701 24292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:02.045567 24359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:02.047655 24359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2: Bootstrap starting.
I20260927 18:38:02.048173 24359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:02.048446 24359 log.cc:824] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:02.049073 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 24292
I20260927 18:38:02.049089 24359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2: No bootstrap required, opened a new log
I20260927 18:38:02.049196 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:02.049377 5727 external_mini_cluster.cc:1428] Setting key f061a564c6dc0c2a7bd2415cb1689979
I20260927 18:38:02.051357 24359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a366d102b74f73a83d7951be572ff2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 46505 } }
I20260927 18:38:02.051522 24359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:02.051561 24359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a366d102b74f73a83d7951be572ff2, State: Initialized, Role: FOLLOWER
I20260927 18:38:02.051713 24359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a366d102b74f73a83d7951be572ff2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 46505 } }
I20260927 18:38:02.051784 24359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:02.051811 24359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:02.051854 24359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:02.052451 24359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a366d102b74f73a83d7951be572ff2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 46505 } }
I20260927 18:38:02.052543 24359 leader_election.cc:302] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34a366d102b74f73a83d7951be572ff2; no voters:
I20260927 18:38:02.052670 24359 leader_election.cc:288] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:02.052716 24364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:02.052850 24364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [term 1 LEADER]: Becoming Leader. State: Replica: 34a366d102b74f73a83d7951be572ff2, State: Running, Role: LEADER
I20260927 18:38:02.052951 24364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a366d102b74f73a83d7951be572ff2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 46505 } }
I20260927 18:38:02.052857 24359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:02.054185 24365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34a366d102b74f73a83d7951be572ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a366d102b74f73a83d7951be572ff2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 46505 } } }
I20260927 18:38:02.054268 24365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:02.054378 24366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a366d102b74f73a83d7951be572ff2. Latest consensus state: current_term: 1 leader_uuid: "34a366d102b74f73a83d7951be572ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a366d102b74f73a83d7951be572ff2" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 46505 } } }
I20260927 18:38:02.054709 24366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2 [sys.catalog]: This master's current role is: LEADER
W20260927 18:38:02.054983 24379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:02.055045 24379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:02.055102 24380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:02.055279 24380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:02.056172 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.049845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.5.151.254:46505)
0927 18:38:02.050075 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.050082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.050112 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:38:02.050912 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.050920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.050992 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.051134 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:02.051326 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.052874 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.053982 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.054607 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.054706 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.055505 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:02.055509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:02.055509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:02.055700 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:02.055738 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:02.055751 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:02.055817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.055909 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.055978 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260927 18:38:02.056932 24380 catalog_manager.cc:1347] Generated new cluster ID: 64312d73fa2b49fda6fbac253bc1790f
I20260927 18:38:02.056986 24380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:02.093079 24380 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:02.093667 24380 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:02.104246 24380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2: Generated new TSK 0
I20260927 18:38:02.104391 24380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:02.116707 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:46505
--builtin_ntp_servers=127.5.151.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:38:02.193006 24383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:02.193246 24383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:02.193279 24383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:02.194361 24383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:02.194419 24383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:02.194459 24383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:38:02.195737 24383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:02.195976 24383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:02.196307 24383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:38:02.200935 24383 server_base.cc:1034] running on GCE node
W20260927 18:38:02.200852 24389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:02.200829 24391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:02.200837 24388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:02.201285 24383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:02.201519 24383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:02.202664 24383 hybrid_clock.cc:648] HybridClock initialized: now 1790534282202651 us; error 26 us; skew 500 ppm
I20260927 18:38:02.207135 24383 webserver.cc:460] Webserver started at http://127.5.151.193:33699/ using document root <none> and password file <none>
I20260927 18:38:02.207386 24383 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:02.207453 24383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:02.207598 24383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:02.208612 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "92e9ba113a814510831f8e8a779201c2"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "29b3c26c7bb6d6654e3c1cf1d151673b"
server_key_iv: "67fc3b315f2076dc90ef8698499540e9"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.208956 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "92e9ba113a814510831f8e8a779201c2"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "29b3c26c7bb6d6654e3c1cf1d151673b"
server_key_iv: "67fc3b315f2076dc90ef8698499540e9"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.210163 24383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 18:38:02.210860 24397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.211019 24383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.211112 24383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "92e9ba113a814510831f8e8a779201c2"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "29b3c26c7bb6d6654e3c1cf1d151673b"
server_key_iv: "67fc3b315f2076dc90ef8698499540e9"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.211191 24383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:02.228832 24383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:02.229135 24383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:02.229274 24383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:02.229483 24383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:02.229787 24383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:02.229839 24383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.229879 24383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:02.229907 24383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.235541 24383 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:43343
I20260927 18:38:02.235576 24510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:43343 every 8 connection(s)
I20260927 18:38:02.235903 24383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:02.241195 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.236602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:44703 (local address 127.5.151.254:46505)
0927 18:38:02.236667 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.236669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.236934 (+ 265us) server_negotiation.cc:407] Connection header received
0927 18:38:02.237422 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.237423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.237432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.237484 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:02.238050 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.238548 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.240068 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.240265 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.240319 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.240985 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:02.240986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:02.240986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:02.240990 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:02.241002 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:02.241005 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:02.241046 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.241108 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.241132 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:38:02.241411 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.236600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:46505 (local address 127.5.151.193:44703)
0927 18:38:02.236771 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.236837 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.237338 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.237533 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.237536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.237543 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:02.237988 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.237992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.238673 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.238673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.239990 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.239994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.240335 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.240335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.240435 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.240870 (+ 435us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:02.240870 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:02.240895 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:02.240914 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:02.241057 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:02.241061 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:02.241063 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.241110 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.241185 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260927 18:38:02.241503 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 24383
I20260927 18:38:02.241606 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:02.241811 5727 external_mini_cluster.cc:1428] Setting key 0399e846519cfc4f641636dbfb7b4d11
I20260927 18:38:02.241889 24511 heartbeater.cc:347] Connected to a master server at 127.5.151.254:46505
I20260927 18:38:02.242094 24511 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:02.242273 24511 heartbeater.cc:511] Master 127.5.151.254:46505 requested a full tablet report, sending...
I20260927 18:38:02.242769 24323 ts_manager.cc:194] Registered new tserver with Master: 92e9ba113a814510831f8e8a779201c2 (127.5.151.193:43343)
I20260927 18:38:02.242797 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:46505
--builtin_ntp_servers=127.5.151.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:38:02.243377 24323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.193:44703
W20260927 18:38:02.319929 24514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:02.320109 24514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:02.320130 24514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:02.321115 24514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:02.321156 24514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:02.321179 24514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:38:02.322348 24514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:02.322516 24514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:02.322788 24514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:02.327594 24519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:02.327554 24522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:02.327772 24514 server_base.cc:1034] running on GCE node
W20260927 18:38:02.327829 24520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:02.328019 24514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:02.328198 24514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:02.329372 24514 hybrid_clock.cc:648] HybridClock initialized: now 1790534282329361 us; error 41 us; skew 500 ppm
I20260927 18:38:02.330703 24514 webserver.cc:460] Webserver started at http://127.5.151.194:36383/ using document root <none> and password file <none>
I20260927 18:38:02.330931 24514 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:02.331027 24514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:02.331171 24514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:02.332217 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "78a2ef6699e44e4cbdaf5db84d8972af"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "51de325fad8c5aa9372c74fbc52edb7b"
server_key_iv: "3c162da244c315821e07c6471bab1ca9"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.332566 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "78a2ef6699e44e4cbdaf5db84d8972af"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "51de325fad8c5aa9372c74fbc52edb7b"
server_key_iv: "3c162da244c315821e07c6471bab1ca9"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.333835 24514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:02.334504 24528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.334743 24514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:02.334839 24514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "78a2ef6699e44e4cbdaf5db84d8972af"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "51de325fad8c5aa9372c74fbc52edb7b"
server_key_iv: "3c162da244c315821e07c6471bab1ca9"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.334889 24514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:02.345242 24514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:02.345434 24514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:02.345522 24514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:02.345679 24514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:02.345903 24514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:02.345929 24514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.345948 24514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:02.345960 24514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.351295 24514 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:44297
I20260927 18:38:02.351333 24641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:44297 every 8 connection(s)
I20260927 18:38:02.351643 24514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:02.356482 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.352308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:36229 (local address 127.5.151.254:46505)
0927 18:38:02.352383 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.352385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.352643 (+ 258us) server_negotiation.cc:407] Connection header received
0927 18:38:02.353109 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.353110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.353119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.353204 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:02.353933 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.354439 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.355426 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.355596 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.355652 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.356281 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:02.356282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:02.356283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:02.356286 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:02.356297 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:02.356301 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:02.356345 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.356406 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.356427 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:38:02.356642 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.352269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:46505 (local address 127.5.151.194:36229)
0927 18:38:02.352530 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.352563 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.352969 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.353270 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.353292 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.353302 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:02.353863 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.353867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.354527 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.354528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.355356 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.355359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.355666 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.355667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.355764 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.356168 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:02.356169 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:02.356193 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:02.356212 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:02.356348 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:02.356349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:02.356351 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.356394 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.356457 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260927 18:38:02.357129 24642 heartbeater.cc:347] Connected to a master server at 127.5.151.254:46505
I20260927 18:38:02.357158 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 24514
I20260927 18:38:02.357225 24642 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:02.357252 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:02.357405 24642 heartbeater.cc:511] Master 127.5.151.254:46505 requested a full tablet report, sending...
I20260927 18:38:02.357438 5727 external_mini_cluster.cc:1428] Setting key 7bf4187587a670831d065ed1ef04f151
I20260927 18:38:02.357859 24323 ts_manager.cc:194] Registered new tserver with Master: 78a2ef6699e44e4cbdaf5db84d8972af (127.5.151.194:44297)
I20260927 18:38:02.358363 24323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.194:36229
I20260927 18:38:02.358649 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:46505
--builtin_ntp_servers=127.5.151.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:38:02.434722 24645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:02.434887 24645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:02.434906 24645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:02.435869 24645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:02.435904 24645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:02.435926 24645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:38:02.437202 24645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:02.437402 24645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:02.437610 24645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:02.442145 24653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:02.442271 24645 server_base.cc:1034] running on GCE node
W20260927 18:38:02.442162 24651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:02.442348 24650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:02.442543 24645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:02.442811 24645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:02.443940 24645 hybrid_clock.cc:648] HybridClock initialized: now 1790534282443931 us; error 26 us; skew 500 ppm
I20260927 18:38:02.445035 24645 webserver.cc:460] Webserver started at http://127.5.151.195:36249/ using document root <none> and password file <none>
I20260927 18:38:02.445259 24645 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:02.445302 24645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:02.445436 24645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:02.446400 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "3676a2834086470cb265f76e12bed8f8"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "b4ae54506ead500e89d78ea4cc850fd5"
server_key_iv: "0a776365d53003450c9376531ee4cbe6"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.446762 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "3676a2834086470cb265f76e12bed8f8"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "b4ae54506ead500e89d78ea4cc850fd5"
server_key_iv: "0a776365d53003450c9376531ee4cbe6"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.447968 24645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:38:02.448592 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.448758 24645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:02.448848 24645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "3676a2834086470cb265f76e12bed8f8"
format_stamp: "Formatted at 2026-09-27 18:38:02 on dist-test-slave-19nl"
server_key: "b4ae54506ead500e89d78ea4cc850fd5"
server_key_iv: "0a776365d53003450c9376531ee4cbe6"
server_key_version: "encryptionkey@0"
I20260927 18:38:02.448925 24645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:02.459407 24645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:02.459657 24645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:02.459789 24645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:02.459975 24645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:02.460242 24645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:02.460289 24645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.460326 24645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:02.460350 24645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:02.465687 24645 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:35873
I20260927 18:38:02.465721 24772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:35873 every 8 connection(s)
I20260927 18:38:02.466015 24645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:02.470340 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.466724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:55819 (local address 127.5.151.254:46505)
0927 18:38:02.466764 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.466765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.466976 (+ 211us) server_negotiation.cc:407] Connection header received
0927 18:38:02.467405 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.467406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.467416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.467477 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:02.468002 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.468432 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.469290 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.469457 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.469503 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.470153 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:02.470154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:02.470154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:02.470158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:02.470170 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:02.470174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:02.470213 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.470268 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.470286 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:38:02.470520 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.466732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:46505 (local address 127.5.151.195:55819)
0927 18:38:02.466874 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.466901 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.467315 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.467522 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.467525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.467532 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:02.467944 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.467947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.468497 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.468498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.469224 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.469225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.469521 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.469522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.469615 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.470039 (+ 424us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:02.470039 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:02.470062 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:02.470082 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:02.470212 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:02.470214 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:02.470217 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.470260 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.470324 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":75,"threads_started":1}
I20260927 18:38:02.471099 24773 heartbeater.cc:347] Connected to a master server at 127.5.151.254:46505
I20260927 18:38:02.471199 24773 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:02.471395 24773 heartbeater.cc:511] Master 127.5.151.254:46505 requested a full tablet report, sending...
I20260927 18:38:02.471796 24323 ts_manager.cc:194] Registered new tserver with Master: 3676a2834086470cb265f76e12bed8f8 (127.5.151.195:35873)
I20260927 18:38:02.472301 24323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.195:55819
I20260927 18:38:02.473057 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 24645
I20260927 18:38:02.473201 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:02.473451 5727 external_mini_cluster.cc:1428] Setting key 9e847e7a44877a24a3fda48ee6af25ff
I20260927 18:38:02.474685 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:02.480821 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.477735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.5.151.254:46505)
0927 18:38:02.477799 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.477801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.477875 (+ 74us) server_negotiation.cc:407] Connection header received
0927 18:38:02.477964 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.477965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.477976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.478035 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:02.478355 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.479136 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.479878 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.480040 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.480087 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.480583 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:02.480585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:02.480585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:02.480588 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:02.480599 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:02.480602 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:02.480667 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.480708 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.480716 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:38:02.482121 24323 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:02.483554 24323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:02.489231 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.485457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:44297 (local address 127.0.0.1:58012)
0927 18:38:02.485811 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.485836 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.485884 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.486116 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.486118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.486121 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:02.486180 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.486182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.487168 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.487168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.488033 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.488035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.488967 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.488968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.489014 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.489021 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.489110 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.489157 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":74,"threads_started":1}
I20260927 18:38:02.489243 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.485633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.5.151.194:44297)
0927 18:38:02.485948 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.485952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.485961 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:02.486003 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.486004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.486016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.486110 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:02.486290 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.487093 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.488126 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.488822 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.488858 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.488926 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.489124 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.489167 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":84,"threads_started":1}
I20260927 18:38:02.489496 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.485457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:35873 (local address 127.0.0.1:56416)
0927 18:38:02.485953 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.485959 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.486020 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.486190 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.486190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.486193 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:02.486248 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.486250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.487580 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.487581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.488448 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.488450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.489271 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.489272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.489338 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.489342 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.489396 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.489430 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":18,"thread_start_us":42,"threads_started":1}
I20260927 18:38:02.489706 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.485513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.5.151.195:35873)
0927 18:38:02.485657 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.485662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.486020 (+ 358us) server_negotiation.cc:407] Connection header received
0927 18:38:02.486083 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.486085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.486101 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.486192 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:02.486420 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.487489 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.488533 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.489094 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.489168 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.489239 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.489601 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.489640 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260927 18:38:02.489962 24576 tablet_service.cc:1467] Processing CreateTablet for tablet b1384f28902346c69ff99edfde603967 (DEFAULT_TABLE table=test-table [id=d64f04cf2c3c431c8dd1fa8285f51403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:02.490192 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.485652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43343 (local address 127.0.0.1:54612)
0927 18:38:02.486210 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.486231 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.486278 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.486524 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.486525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.486526 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:02.486558 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.486583 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.487743 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.487744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.488748 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.488751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.489977 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.489978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.490032 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.490036 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.490087 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.490130 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":69,"threads_started":1}
I20260927 18:38:02.490211 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.485741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.5.151.193:43343)
0927 18:38:02.485895 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.485900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.486339 (+ 439us) server_negotiation.cc:407] Connection header received
0927 18:38:02.486382 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.486384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.486398 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.486461 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:02.486643 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.487665 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.488844 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.489881 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.489974 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.490053 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.490102 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.490136 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260927 18:38:02.490291 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1384f28902346c69ff99edfde603967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:02.490259 24707 tablet_service.cc:1467] Processing CreateTablet for tablet b1384f28902346c69ff99edfde603967 (DEFAULT_TABLE table=test-table [id=d64f04cf2c3c431c8dd1fa8285f51403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:02.490530 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1384f28902346c69ff99edfde603967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:02.494053 24788 tablet_bootstrap.cc:492] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Bootstrap starting.
I20260927 18:38:02.494315 24445 tablet_service.cc:1467] Processing CreateTablet for tablet b1384f28902346c69ff99edfde603967 (DEFAULT_TABLE table=test-table [id=d64f04cf2c3c431c8dd1fa8285f51403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:02.494674 24445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1384f28902346c69ff99edfde603967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:02.494686 24787 tablet_bootstrap.cc:492] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Bootstrap starting.
I20260927 18:38:02.494737 24788 tablet_bootstrap.cc:654] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:02.495030 24788 log.cc:824] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:02.495251 24787 tablet_bootstrap.cc:654] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:02.495561 24787 log.cc:824] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:02.495780 24788 tablet_bootstrap.cc:492] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: No bootstrap required, opened a new log
I20260927 18:38:02.495852 24788 ts_tablet_manager.cc:1397] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:38:02.496248 24787 tablet_bootstrap.cc:492] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: No bootstrap required, opened a new log
I20260927 18:38:02.496326 24787 ts_tablet_manager.cc:1397] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:38:02.498000 24791 tablet_bootstrap.cc:492] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2: Bootstrap starting.
I20260927 18:38:02.498371 24788 raft_consensus.cc:348] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.498520 24788 raft_consensus.cc:374] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:02.498574 24788 raft_consensus.cc:729] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3676a2834086470cb265f76e12bed8f8, State: Initialized, Role: FOLLOWER
I20260927 18:38:02.498721 24791 tablet_bootstrap.cc:654] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:02.498710 24788 consensus_queue.cc:260] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.498659 24787 raft_consensus.cc:348] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.498805 24787 raft_consensus.cc:374] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:02.498844 24787 raft_consensus.cc:729] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a2ef6699e44e4cbdaf5db84d8972af, State: Initialized, Role: FOLLOWER
I20260927 18:38:02.498931 24788 ts_tablet_manager.cc:1428] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:02.498974 24787 consensus_queue.cc:260] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.499010 24791 log.cc:824] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:02.499032 24773 heartbeater.cc:503] Master 127.5.151.254:46505 was elected leader, sending a full tablet report...
I20260927 18:38:02.499226 24787 ts_tablet_manager.cc:1428] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:02.499256 24642 heartbeater.cc:503] Master 127.5.151.254:46505 was elected leader, sending a full tablet report...
I20260927 18:38:02.499802 24791 tablet_bootstrap.cc:492] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2: No bootstrap required, opened a new log
I20260927 18:38:02.499863 24791 ts_tablet_manager.cc:1397] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:38:02.501744 24791 raft_consensus.cc:348] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.501854 24791 raft_consensus.cc:374] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:02.501873 24791 raft_consensus.cc:729] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e9ba113a814510831f8e8a779201c2, State: Initialized, Role: FOLLOWER
I20260927 18:38:02.501968 24791 consensus_queue.cc:260] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.502166 24791 ts_tablet_manager.cc:1428] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 18:38:02.502239 24511 heartbeater.cc:503] Master 127.5.151.254:46505 was elected leader, sending a full tablet report...
W20260927 18:38:02.602200 24643 tablet.cc:2369] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:02.716517 24774 tablet.cc:2369] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:02.736552 24512 tablet.cc:2369] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:02.791930 24792 raft_consensus.cc:482] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:02.792083 24792 raft_consensus.cc:504] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.792443 24792 leader_election.cc:288] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78a2ef6699e44e4cbdaf5db84d8972af (127.5.151.194:44297), 92e9ba113a814510831f8e8a779201c2 (127.5.151.193:43343)
I20260927 18:38:02.795312 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.792738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:48785 (local address 127.5.151.194:44297)
0927 18:38:02.792782 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.792784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.792796 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:38:02.792832 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.792833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.792843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.792911 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:02.793156 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.793676 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.794632 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.794984 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.795065 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.795073 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.795204 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.795247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:38:02.795302 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.792656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:44297 (local address 127.5.151.195:48785)
0927 18:38:02.792715 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.792727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.792770 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.792930 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.792931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.792933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:02.793049 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.793051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.793754 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.793755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.794510 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.794512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.795064 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.795065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.795137 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.795141 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.795181 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.795210 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":25}
I20260927 18:38:02.795379 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.792874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:44435 (local address 127.5.151.193:43343)
0927 18:38:02.792923 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.792924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.793013 (+ 89us) server_negotiation.cc:407] Connection header received
0927 18:38:02.793055 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.793056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.793068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.793136 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:02.793286 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.793733 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.794658 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.794989 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.795071 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.795078 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.795273 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.795310 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:38:02.795408 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.792769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43343 (local address 127.5.151.195:44435)
0927 18:38:02.792936 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.792946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:02.792999 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:02.793141 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:02.793141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:02.793143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:02.793217 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.793219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.793821 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.793822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.794533 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:02.794535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.795093 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.795094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:02.795191 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.795195 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:38:02.795249 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:38:02.795283 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260927 18:38:02.795665 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1384f28902346c69ff99edfde603967" candidate_uuid: "3676a2834086470cb265f76e12bed8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" is_pre_election: true
I20260927 18:38:02.795660 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1384f28902346c69ff99edfde603967" candidate_uuid: "3676a2834086470cb265f76e12bed8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e9ba113a814510831f8e8a779201c2" is_pre_election: true
I20260927 18:38:02.795805 24596 raft_consensus.cc:2454] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3676a2834086470cb265f76e12bed8f8 in term 0.
I20260927 18:38:02.795805 24465 raft_consensus.cc:2454] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3676a2834086470cb265f76e12bed8f8 in term 0.
I20260927 18:38:02.796036 24660 leader_election.cc:302] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3676a2834086470cb265f76e12bed8f8, 92e9ba113a814510831f8e8a779201c2; no voters:
I20260927 18:38:02.796164 24792 raft_consensus.cc:2781] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:02.796214 24792 raft_consensus.cc:482] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:02.796238 24792 raft_consensus.cc:3037] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:02.796876 24792 raft_consensus.cc:504] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.796993 24792 leader_election.cc:288] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [CANDIDATE]: Term 1 election: Requested vote from peers 78a2ef6699e44e4cbdaf5db84d8972af (127.5.151.194:44297), 92e9ba113a814510831f8e8a779201c2 (127.5.151.193:43343)
I20260927 18:38:02.797187 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1384f28902346c69ff99edfde603967" candidate_uuid: "3676a2834086470cb265f76e12bed8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e9ba113a814510831f8e8a779201c2"
I20260927 18:38:02.797197 24596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1384f28902346c69ff99edfde603967" candidate_uuid: "3676a2834086470cb265f76e12bed8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a2ef6699e44e4cbdaf5db84d8972af"
I20260927 18:38:02.797273 24596 raft_consensus.cc:3037] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:02.797273 24465 raft_consensus.cc:3037] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:02.798137 24596 raft_consensus.cc:2454] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3676a2834086470cb265f76e12bed8f8 in term 1.
I20260927 18:38:02.798163 24465 raft_consensus.cc:2454] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3676a2834086470cb265f76e12bed8f8 in term 1.
I20260927 18:38:02.798333 24660 leader_election.cc:302] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3676a2834086470cb265f76e12bed8f8, 78a2ef6699e44e4cbdaf5db84d8972af; no voters:
I20260927 18:38:02.798446 24792 raft_consensus.cc:2781] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:02.798609 24792 raft_consensus.cc:686] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3676a2834086470cb265f76e12bed8f8, State: Running, Role: LEADER
I20260927 18:38:02.798735 24792 consensus_queue.cc:237] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } }
I20260927 18:38:02.799590 24322 catalog_manager.cc:5495] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3676a2834086470cb265f76e12bed8f8 (127.5.151.195). New cstate: current_term: 1 leader_uuid: "3676a2834086470cb265f76e12bed8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3676a2834086470cb265f76e12bed8f8" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 35873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:02.839977 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.837590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.5.151.195:35873)
0927 18:38:02.837653 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:02.837655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:02.837664 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:02.837730 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:02.837731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:02.837750 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:02.837826 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:02.837960 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.838476 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.839252 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:02.839436 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:02.839508 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:02.839642 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:02.839734 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:02.839842 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:02.839871 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 18:38:02.839879 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:38:02.841621 24596 raft_consensus.cc:1261] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 1 FOLLOWER]: Refusing update from remote peer 3676a2834086470cb265f76e12bed8f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:02.841615 24465 raft_consensus.cc:1261] T b1384f28902346c69ff99edfde603967 P 92e9ba113a814510831f8e8a779201c2 [term 1 FOLLOWER]: Refusing update from remote peer 3676a2834086470cb265f76e12bed8f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:02.842005 24792 consensus_queue.cc:1035] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e9ba113a814510831f8e8a779201c2" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:02.842144 24797 consensus_queue.cc:1035] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a2ef6699e44e4cbdaf5db84d8972af" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:02.844313 24800 mvcc.cc:205] Tried to move back new op lower bound from 7334028422516768768 to 7334028422344237056. Current Snapshot: MvccSnapshot[applied={T|T < 7334028422516768768}]
I20260927 18:38:02.844738 24802 mvcc.cc:205] Tried to move back new op lower bound from 7334028422516768768 to 7334028422344237056. Current Snapshot: MvccSnapshot[applied={T|T < 7334028422516768768}]
I20260927 18:38:02.844915 24801 mvcc.cc:205] Tried to move back new op lower bound from 7334028422516768768 to 7334028422344237056. Current Snapshot: MvccSnapshot[applied={T|T < 7334028422516768768 or (T in {7334028422516768768})}]
I20260927 18:38:02.846634 24322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:02.846733 24322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:02.847420 24322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34a366d102b74f73a83d7951be572ff2: Sending DeleteTablet for 3 replicas of tablet b1384f28902346c69ff99edfde603967
I20260927 18:38:02.847703 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet b1384f28902346c69ff99edfde603967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:02 UTC) from {username='slave'} at 127.0.0.1:58012
I20260927 18:38:02.847761 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet b1384f28902346c69ff99edfde603967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:02 UTC) from {username='slave'} at 127.0.0.1:56416
I20260927 18:38:02.847903 24445 tablet_service.cc:1514] Processing DeleteTablet for tablet b1384f28902346c69ff99edfde603967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:02 UTC) from {username='slave'} at 127.0.0.1:54612
I20260927 18:38:02.847947 24811 tablet_replica.cc:330] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: stopping tablet replica
I20260927 18:38:02.848033 24811 raft_consensus.cc:2229] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:02.848125 24811 raft_consensus.cc:2258] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:02.848335 24810 tablet_replica.cc:330] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: stopping tablet replica
I20260927 18:38:02.848409 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 24383
I20260927 18:38:02.848423 24810 raft_consensus.cc:2229] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:02.848510 24810 raft_consensus.cc:2258] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:02.849107 24811 ts_tablet_manager.cc:1905] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:02.850627 24810 ts_tablet_manager.cc:1905] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:02.852083 24810 ts_tablet_manager.cc:1918] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:02.852160 24810 log.cc:1196] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/b1384f28902346c69ff99edfde603967
I20260927 18:38:02.852409 24810 ts_tablet_manager.cc:1939] T b1384f28902346c69ff99edfde603967 P 78a2ef6699e44e4cbdaf5db84d8972af: Deleting consensus metadata
I20260927 18:38:02.852789 24307 catalog_manager.cc:4841] TS 78a2ef6699e44e4cbdaf5db84d8972af (127.5.151.194:44297): tablet b1384f28902346c69ff99edfde603967 (table test-table [id=d64f04cf2c3c431c8dd1fa8285f51403]) successfully deleted
I20260927 18:38:02.853188 24811 ts_tablet_manager.cc:1918] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:02.853271 24811 log.cc:1196] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/b1384f28902346c69ff99edfde603967
I20260927 18:38:02.853550 24811 ts_tablet_manager.cc:1939] T b1384f28902346c69ff99edfde603967 P 3676a2834086470cb265f76e12bed8f8: Deleting consensus metadata
I20260927 18:38:02.853905 24309 catalog_manager.cc:4841] TS 3676a2834086470cb265f76e12bed8f8 (127.5.151.195:35873): tablet b1384f28902346c69ff99edfde603967 (table test-table [id=d64f04cf2c3c431c8dd1fa8285f51403]) successfully deleted
W20260927 18:38:02.855326 24307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:43343 (error 108)
I20260927 18:38:02.855782 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:02.855645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:43343 (local address 127.0.0.1:54622)
0927 18:38:02.855705 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:02.855745 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 18:38:02.855844 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 24514
W20260927 18:38:02.859421 24307 catalog_manager.cc:4584] TS 92e9ba113a814510831f8e8a779201c2 (127.5.151.193:43343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1384f28902346c69ff99edfde603967: Network error: Client connection negotiation failed: client connection to 127.5.151.193:43343: connect: Connection refused (error 111)
I20260927 18:38:02.863283 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 24645
I20260927 18:38:02.867561 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 24292
2026-09-27T18:38:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (957 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:02 dist-test-slave-19nl krb5kdc[24816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:02 dist-test-slave-19nl krb5kdc[24816](info): set up 2 sockets
Sep 27 18:38:02 dist-test-slave-19nl krb5kdc[24816](info): commencing operation
krb5kdc: starting...
W20260927 18:38:03.850020 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:03 dist-test-slave-19nl krb5kdc[24816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:03.955116 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38193
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:41529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:38193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:04.033623 24829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:04.033820 24829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:04.033839 24829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:04.034993 24829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:04.035033 24829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:04.035048 24829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:04.035059 24829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:04.035071 24829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:04.036449 24829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:41529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:38193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:38193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:04.036677 24829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:04.036886 24829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:04.041258 24835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.041260 24837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.041277 24834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.041527 24829 server_base.cc:1034] running on GCE node
I20260927 18:38:04.041787 24829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:04.042014 24829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:04.043146 24829 hybrid_clock.cc:648] HybridClock initialized: now 1790534284043134 us; error 27 us; skew 500 ppm
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534284, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:04.045656 24829 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:04.046039 24829 webserver.cc:460] Webserver started at http://127.5.151.254:42249/ using document root <none> and password file <none>
I20260927 18:38:04.046213 24829 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:04.046249 24829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:04.046329 24829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:04.047246 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "51fa0df916b64f57879610ca2c9b8205"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "415747d4188bfff5987cb719d1a6d6d8"
server_key_iv: "619251b17426077772eec25b7673179c"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.047513 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "51fa0df916b64f57879610ca2c9b8205"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "415747d4188bfff5987cb719d1a6d6d8"
server_key_iv: "619251b17426077772eec25b7673179c"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.048599 24829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:04.049226 24844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.049381 24829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:04.049463 24829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "51fa0df916b64f57879610ca2c9b8205"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "415747d4188bfff5987cb719d1a6d6d8"
server_key_iv: "619251b17426077772eec25b7673179c"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.049525 24829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:04.056597 24829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:04.058694 24829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:04.058836 24829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:04.062397 24829 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:38193
I20260927 18:38:04.062423 24896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:38193 every 8 connection(s)
I20260927 18:38:04.062745 24829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:04.063541 24897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:04.065620 24897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205: Bootstrap starting.
I20260927 18:38:04.066174 24897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:04.066438 24897 log.cc:824] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:04.067098 24897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205: No bootstrap required, opened a new log
I20260927 18:38:04.068497 24897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fa0df916b64f57879610ca2c9b8205" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38193 } }
I20260927 18:38:04.068637 24897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:04.068681 24897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fa0df916b64f57879610ca2c9b8205, State: Initialized, Role: FOLLOWER
I20260927 18:38:04.068830 24897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fa0df916b64f57879610ca2c9b8205" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38193 } }
I20260927 18:38:04.068935 24897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:04.068979 24897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:04.069028 24897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:04.069048 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 24829
I20260927 18:38:04.069164 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:04.069357 5727 external_mini_cluster.cc:1428] Setting key 6b7d6dfe32a1d5dfb2569d33fb8cfcf2
I20260927 18:38:04.069653 24897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fa0df916b64f57879610ca2c9b8205" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38193 } }
I20260927 18:38:04.069761 24897 leader_election.cc:302] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fa0df916b64f57879610ca2c9b8205; no voters:
I20260927 18:38:04.069903 24897 leader_election.cc:288] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:04.069947 24902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:04.070144 24902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [term 1 LEADER]: Becoming Leader. State: Replica: 51fa0df916b64f57879610ca2c9b8205, State: Running, Role: LEADER
I20260927 18:38:04.070192 24897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:04.070262 24902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fa0df916b64f57879610ca2c9b8205" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38193 } }
I20260927 18:38:04.070727 24902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fa0df916b64f57879610ca2c9b8205. Latest consensus state: current_term: 1 leader_uuid: "51fa0df916b64f57879610ca2c9b8205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fa0df916b64f57879610ca2c9b8205" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38193 } } }
I20260927 18:38:04.070863 24902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:04.070894 24903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51fa0df916b64f57879610ca2c9b8205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fa0df916b64f57879610ca2c9b8205" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 38193 } } }
I20260927 18:38:04.070968 24903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:04.071163 24911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:04.071607 24911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:04.073148 24911 catalog_manager.cc:1347] Generated new cluster ID: 3c196d1e634b424383d8d0f90c03a90d
I20260927 18:38:04.073199 24911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:04.078797 24911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:04.079384 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.069766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.5.151.254:38193)
0927 18:38:04.069923 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.069933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.070027 (+ 94us) server_negotiation.cc:407] Connection header received
0927 18:38:04.070181 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.070189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.070241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.071142 (+ 901us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:04.071249 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.072527 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.073438 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.073852 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.073984 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.076211 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:04.076215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:04.076215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:04.076225 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:04.077685 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.078029 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.078031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.078031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.078133 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.078328 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.078329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.078329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.078618 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:04.078758 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.079150 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.079260 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260927 18:38:04.079396 24911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:04.085655 24911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205: Generated new TSK 0
I20260927 18:38:04.085791 24911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.151.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
Entry for principal kudu/127.5.151.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab.
I20260927 18:38:04.111465 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:41529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:04.187963 24923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:04.188162 24923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:04.188184 24923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:04.189281 24923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:04.189319 24923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:04.189343 24923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:38:04.190783 24923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:41529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.193.keytab
--principal=kudu/127.5.151.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:04.191001 24923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:04.191250 24923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:04.196017 24931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.196115 24929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.196240 24928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.196429 24923 server_base.cc:1034] running on GCE node
I20260927 18:38:04.196636 24923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:04.196887 24923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:04.198098 24923 hybrid_clock.cc:648] HybridClock initialized: now 1790534284198085 us; error 26 us; skew 500 ppm
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534284, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:04.200572 24923 init.cc:377] Logged in from keytab as kudu/127.5.151.193@KRBTEST.COM (short username kudu)
I20260927 18:38:04.200927 24923 webserver.cc:460] Webserver started at http://127.5.151.193:44087/ using document root <none> and password file <none>
I20260927 18:38:04.201093 24923 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:04.201138 24923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:04.201218 24923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:04.202145 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "21ca9801991645d28c70ab2c135f3be0"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "8e6387d221ca808d86ef911e1100962d"
server_key_iv: "c564941bb83af3a260c4dbe8d563c3c0"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.202407 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "21ca9801991645d28c70ab2c135f3be0"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "8e6387d221ca808d86ef911e1100962d"
server_key_iv: "c564941bb83af3a260c4dbe8d563c3c0"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.203608 24923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:04.204248 24938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.204418 24923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:04.204500 24923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "21ca9801991645d28c70ab2c135f3be0"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "8e6387d221ca808d86ef911e1100962d"
server_key_iv: "c564941bb83af3a260c4dbe8d563c3c0"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.204581 24923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:04.214184 24923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:04.216306 24923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:04.216445 24923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:04.216656 24923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:04.216929 24923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:04.216980 24923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.217017 24923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:04.217043 24923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.222324 24923 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:42115
I20260927 18:38:04.222429 25051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:42115 every 8 connection(s)
I20260927 18:38:04.222724 24923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:04.225591 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 24923
I20260927 18:38:04.225710 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:04.225889 5727 external_mini_cluster.cc:1428] Setting key a449adf80be0aaa7acc5bb343b2abc07
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534284, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.193@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:04.232304 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.223386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:35041 (local address 127.5.151.254:38193)
0927 18:38:04.223453 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.223455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.223585 (+ 130us) server_negotiation.cc:407] Connection header received
0927 18:38:04.223980 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.223981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.223990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.224037 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:04.224814 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.225270 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.226286 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.226516 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.226620 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.229075 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:04.229076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:04.229077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:04.229084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:04.230500 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.231030 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.231031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.231032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.231108 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.231432 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.231433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.231433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.231604 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:04.231677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.232054 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.232194 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:38:04.232381 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.223347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38193 (local address 127.5.151.193:35041)
0927 18:38:04.223480 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.223510 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.223909 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.224090 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.224093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.224587 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:04.224748 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.224752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.225339 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.225340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.226213 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.226216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.226646 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.226647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.226805 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.227222 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:04.227232 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:04.229002 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:04.230606 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:04.230613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:04.230615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:04.230956 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:04.231207 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:04.231208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:04.231209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:04.231352 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:04.231668 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:04.231674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:04.231882 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.232015 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.232172 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
I20260927 18:38:04.232956 25052 heartbeater.cc:347] Connected to a master server at 127.5.151.254:38193
I20260927 18:38:04.233104 25052 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:04.233294 25052 heartbeater.cc:511] Master 127.5.151.254:38193 requested a full tablet report, sending...
I20260927 18:38:04.233855 24861 ts_manager.cc:194] Registered new tserver with Master: 21ca9801991645d28c70ab2c135f3be0 (127.5.151.193:42115)
I20260927 18:38:04.234448 24861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.193@KRBTEST.COM'} at 127.5.151.193:35041
WARNING: no policy specified for kudu/127.5.151.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
Entry for principal kudu/127.5.151.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab.
I20260927 18:38:04.250294 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:41529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:04.326396 25057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:04.326629 25057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:04.326658 25057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:04.327716 25057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:04.327749 25057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:04.327773 25057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:38:04.329098 25057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:41529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.194.keytab
--principal=kudu/127.5.151.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:04.329268 25057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:04.329430 25057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:04.334096 25062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.334131 25057 server_base.cc:1034] running on GCE node
W20260927 18:38:04.334124 25063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.334100 25065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.334448 25057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:04.334710 25057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:04.335880 25057 hybrid_clock.cc:648] HybridClock initialized: now 1790534284335861 us; error 29 us; skew 500 ppm
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534284, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:04.338385 25057 init.cc:377] Logged in from keytab as kudu/127.5.151.194@KRBTEST.COM (short username kudu)
I20260927 18:38:04.338868 25057 webserver.cc:460] Webserver started at http://127.5.151.194:42401/ using document root <none> and password file <none>
I20260927 18:38:04.339087 25057 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:04.339134 25057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:04.339263 25057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:04.340173 25057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "60bc6f5b58224571aa258395cb6f1312"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "4f75fded08e231e8c04e008a65cb4cf7"
server_key_iv: "d3de6fc43bb962fd500a2e5b68e4d220"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.340489 25057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "60bc6f5b58224571aa258395cb6f1312"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "4f75fded08e231e8c04e008a65cb4cf7"
server_key_iv: "d3de6fc43bb962fd500a2e5b68e4d220"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.341811 25057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:04.342475 25072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.342644 25057 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:04.342737 25057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "60bc6f5b58224571aa258395cb6f1312"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "4f75fded08e231e8c04e008a65cb4cf7"
server_key_iv: "d3de6fc43bb962fd500a2e5b68e4d220"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.342816 25057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:04.357360 25057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:04.359169 25057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:04.359319 25057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:04.359524 25057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:04.359804 25057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:04.359858 25057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.359897 25057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:04.359925 25057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.365329 25057 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:43193
I20260927 18:38:04.365370 25185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:43193 every 8 connection(s)
I20260927 18:38:04.365646 25057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534284, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.194@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:04.374025 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.366281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:56953 (local address 127.5.151.254:38193)
0927 18:38:04.366349 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.366351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.366516 (+ 165us) server_negotiation.cc:407] Connection header received
0927 18:38:04.367059 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.367060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.367069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.367139 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:04.367975 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.368457 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.369378 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.369593 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.369653 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.371862 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:04.371863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:04.371864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:04.371871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:04.373001 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.373341 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.373342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.373342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.373390 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.373548 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.373549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.373549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.373642 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:04.373690 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.373902 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.373964 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:38:04.374202 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.366239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38193 (local address 127.5.151.194:56953)
0927 18:38:04.366402 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.366440 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.366975 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.367190 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.367193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.367719 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:04.367905 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.367908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.368529 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.368530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.369307 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.369309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.369672 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.369672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.369779 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.370247 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:04.370254 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:04.371768 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:04.373073 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:04.373079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:04.373080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:04.373279 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:04.373429 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:04.373429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:04.373429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:04.373503 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:04.373696 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:04.373701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:04.373802 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.373876 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.374001 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260927 18:38:04.374390 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 25057
I20260927 18:38:04.374502 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:04.374652 25186 heartbeater.cc:347] Connected to a master server at 127.5.151.254:38193
I20260927 18:38:04.374714 5727 external_mini_cluster.cc:1428] Setting key 655fd7c722c81bc2ea642aa04fe166dd
I20260927 18:38:04.374742 25186 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:04.374928 25186 heartbeater.cc:511] Master 127.5.151.254:38193 requested a full tablet report, sending...
I20260927 18:38:04.375351 24861 ts_manager.cc:194] Registered new tserver with Master: 60bc6f5b58224571aa258395cb6f1312 (127.5.151.194:43193)
I20260927 18:38:04.375697 24861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.194@KRBTEST.COM'} at 127.5.151.194:56953
WARNING: no policy specified for kudu/127.5.151.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
Entry for principal kudu/127.5.151.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab.
I20260927 18:38:04.396672 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38193
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:41529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:04.473625 25191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:04.473807 25191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:04.473829 25191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:04.474938 25191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:04.474974 25191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:04.475018 25191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:38:04.476404 25191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:41529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.195.keytab
--principal=kudu/127.5.151.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:38193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:04.476619 25191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:04.476855 25191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:04.481431 25197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.481426 25196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.481516 25191 server_base.cc:1034] running on GCE node
W20260927 18:38:04.481426 25199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.481783 25191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:04.482048 25191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:04.483201 25191 hybrid_clock.cc:648] HybridClock initialized: now 1790534284483177 us; error 33 us; skew 500 ppm
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534284, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:04.485487 25191 init.cc:377] Logged in from keytab as kudu/127.5.151.195@KRBTEST.COM (short username kudu)
I20260927 18:38:04.485930 25191 webserver.cc:460] Webserver started at http://127.5.151.195:40389/ using document root <none> and password file <none>
I20260927 18:38:04.486142 25191 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:04.486203 25191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:04.486342 25191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:04.487331 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "f97a4ea49b9f4feb9b37df44ea39b705"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "ab58b12ebe7d7b6ead6165095c521a54"
server_key_iv: "79a9883622a1a1c2dbb2c41176048bb0"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.487674 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "f97a4ea49b9f4feb9b37df44ea39b705"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "ab58b12ebe7d7b6ead6165095c521a54"
server_key_iv: "79a9883622a1a1c2dbb2c41176048bb0"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.488886 25191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:38:04.489511 25206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.489672 25191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:04.489776 25191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "f97a4ea49b9f4feb9b37df44ea39b705"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "ab58b12ebe7d7b6ead6165095c521a54"
server_key_iv: "79a9883622a1a1c2dbb2c41176048bb0"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.489853 25191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:04.504065 25191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:04.505787 25191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:04.505944 25191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:04.506170 25191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:04.506492 25191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:04.506546 25191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.506654 25191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:04.506687 25191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.512583 25191 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:33391
I20260927 18:38:04.512622 25319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:33391 every 8 connection(s)
I20260927 18:38:04.512948 25191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:38:04 dist-test-slave-19nl krb5kdc[24816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534284, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.195@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:04.520946 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 25191
I20260927 18:38:04.521059 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:04.521252 5727 external_mini_cluster.cc:1428] Setting key 81729b0494575144874b4f237678307e
I20260927 18:38:04.521863 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.513612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:40821 (local address 127.5.151.254:38193)
0927 18:38:04.513695 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.513698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.513844 (+ 146us) server_negotiation.cc:407] Connection header received
0927 18:38:04.514347 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.514348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.514360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.514411 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:04.515304 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.515795 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.516799 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.517013 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.517089 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.519192 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:04.519193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:04.519194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:04.519201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:04.520335 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.520728 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.520730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.520730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.520810 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.521103 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.521104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.521105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.521264 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:04.521350 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.521658 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.521747 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:38:04.522054 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.513565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:38193 (local address 127.5.151.195:40821)
0927 18:38:04.513725 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.513758 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.514242 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.514473 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.514477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.515025 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:04.515207 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.515211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.515912 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.515913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.516706 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.516709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.517095 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.517096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.517222 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.517732 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:04.517740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:04.519092 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:04.520423 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:04.520429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:04.520431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:04.520643 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:04.520881 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:04.520882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:04.520882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:04.520983 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:04.521346 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:04.521352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:04.521510 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.521617 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.521783 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260927 18:38:04.522482 25320 heartbeater.cc:347] Connected to a master server at 127.5.151.254:38193
I20260927 18:38:04.522616 25320 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:04.522773 25320 heartbeater.cc:511] Master 127.5.151.254:38193 requested a full tablet report, sending...
I20260927 18:38:04.523106 24861 ts_manager.cc:194] Registered new tserver with Master: f97a4ea49b9f4feb9b37df44ea39b705 (127.5.151.195:33391)
I20260927 18:38:04.523497 24861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.151.195@KRBTEST.COM'} at 127.5.151.195:40821
I20260927 18:38:04.532950 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:04.539796 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.534278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.5.151.254:38193)
0927 18:38:04.534340 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.534341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.534482 (+ 141us) server_negotiation.cc:407] Connection header received
0927 18:38:04.534519 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.534520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.534528 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.534632 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:04.535149 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.535589 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.536242 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.536407 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.536452 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.537581 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:04.537582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:04.537582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:04.537587 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:04.538806 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.539091 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.539092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.539092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.539148 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:04.539351 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:04.539352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:04.539352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:04.539480 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:04.539550 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.539650 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.539705 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:38:04.541282 24861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:04.543044 24861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:04.549392 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.545342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.5.151.195:33391)
0927 18:38:04.546033 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.546038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.546047 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:04.546093 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.546094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.546109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.546183 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:04.546346 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.547086 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.548544 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.549010 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.549113 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.549265 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.549290 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.549323 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":86,"threads_started":1}
I20260927 18:38:04.549479 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.545463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42115 (local address 127.0.0.1:53898)
0927 18:38:04.545655 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.545665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.545704 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.546126 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.546127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.546130 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:04.546179 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.546180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.547412 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.547413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.548308 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.548310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.549196 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.549198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.549266 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.549273 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.549303 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.549345 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":7,"thread_start_us":73,"threads_started":1}
I20260927 18:38:04.549378 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.545237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:33391 (local address 127.0.0.1:37530)
0927 18:38:04.545600 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.545610 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.545666 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.546256 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.546257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.546259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:04.546293 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.546294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.547193 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.547194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.548464 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.548466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.549115 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.549116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.549186 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.549198 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.549254 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.549303 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260927 18:38:04.549620 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.545701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.5.151.193:42115)
0927 18:38:04.545950 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.545954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.545964 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:38:04.546000 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.546002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.546019 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.546124 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:04.546251 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.547299 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.548389 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.549093 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.549423 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.549492 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.549516 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.549550 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260927 18:38:04.549813 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.545224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:43193 (local address 127.0.0.1:43536)
0927 18:38:04.545433 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.545476 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.545538 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.545887 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.545890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.545893 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:04.545960 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.545962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.547343 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.547344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.548425 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.548427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.549197 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.549198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.549266 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.549283 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.549310 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.549344 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":5,"thread_start_us":85,"threads_started":1}
I20260927 18:38:04.550315 24986 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8cb0f6b4684547a195470ae89b5faf (DEFAULT_TABLE table=test-table [id=43f348748e034d61a732ca70b17885b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:04.550354 25254 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8cb0f6b4684547a195470ae89b5faf (DEFAULT_TABLE table=test-table [id=43f348748e034d61a732ca70b17885b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:04.550683 24986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8cb0f6b4684547a195470ae89b5faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:04.550684 25254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8cb0f6b4684547a195470ae89b5faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:04.553467 25334 tablet_bootstrap.cc:492] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Bootstrap starting.
I20260927 18:38:04.553715 25335 tablet_bootstrap.cc:492] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0: Bootstrap starting.
I20260927 18:38:04.553835 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.545492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.5.151.194:43193)
0927 18:38:04.545698 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.545703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.545714 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:38:04.545777 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.545780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.545797 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.545877 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:04.546049 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.547239 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.548501 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.549116 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.549408 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.553683 (+ 4275us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.553709 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.553750 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260927 18:38:04.554299 25335 tablet_bootstrap.cc:654] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:04.554446 25334 tablet_bootstrap.cc:654] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:04.554534 25120 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8cb0f6b4684547a195470ae89b5faf (DEFAULT_TABLE table=test-table [id=43f348748e034d61a732ca70b17885b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:04.554806 25335 log.cc:824] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:04.554893 25120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8cb0f6b4684547a195470ae89b5faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:04.555226 25334 log.cc:824] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:04.555406 25335 tablet_bootstrap.cc:492] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0: No bootstrap required, opened a new log
I20260927 18:38:04.555490 25335 ts_tablet_manager.cc:1397] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:38:04.556080 25334 tablet_bootstrap.cc:492] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: No bootstrap required, opened a new log
I20260927 18:38:04.556149 25334 ts_tablet_manager.cc:1397] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:38:04.557438 25335 raft_consensus.cc:348] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.557612 25335 raft_consensus.cc:374] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:04.557654 25335 raft_consensus.cc:729] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ca9801991645d28c70ab2c135f3be0, State: Initialized, Role: FOLLOWER
I20260927 18:38:04.557791 25338 tablet_bootstrap.cc:492] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Bootstrap starting.
I20260927 18:38:04.557786 25335 consensus_queue.cc:260] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.558060 25335 ts_tablet_manager.cc:1428] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:04.557991 25334 raft_consensus.cc:348] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.558140 25334 raft_consensus.cc:374] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:04.558183 25334 raft_consensus.cc:729] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97a4ea49b9f4feb9b37df44ea39b705, State: Initialized, Role: FOLLOWER
I20260927 18:38:04.558193 25052 heartbeater.cc:503] Master 127.5.151.254:38193 was elected leader, sending a full tablet report...
I20260927 18:38:04.558295 25334 consensus_queue.cc:260] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.558406 25338 tablet_bootstrap.cc:654] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:04.558543 25334 ts_tablet_manager.cc:1428] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:38:04.558627 25320 heartbeater.cc:503] Master 127.5.151.254:38193 was elected leader, sending a full tablet report...
I20260927 18:38:04.558969 25338 log.cc:824] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:04.559667 25338 tablet_bootstrap.cc:492] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: No bootstrap required, opened a new log
I20260927 18:38:04.559741 25338 ts_tablet_manager.cc:1397] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:38:04.561136 25338 raft_consensus.cc:348] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.561278 25338 raft_consensus.cc:374] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:04.561323 25338 raft_consensus.cc:729] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bc6f5b58224571aa258395cb6f1312, State: Initialized, Role: FOLLOWER
I20260927 18:38:04.561422 25338 consensus_queue.cc:260] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.561663 25338 ts_tablet_manager.cc:1428] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:38:04.561720 25186 heartbeater.cc:503] Master 127.5.151.254:38193 was elected leader, sending a full tablet report...
I20260927 18:38:04.567665 25342 raft_consensus.cc:482] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:04.567754 25342 raft_consensus.cc:504] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.568037 25342 leader_election.cc:288] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f97a4ea49b9f4feb9b37df44ea39b705 (127.5.151.195:33391), 21ca9801991645d28c70ab2c135f3be0 (127.5.151.193:42115)
I20260927 18:38:04.571416 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.568366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42115 (local address 127.5.151.194:36445)
0927 18:38:04.568501 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.568512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.568572 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.568704 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.568705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.568706 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:04.568747 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.568748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.569522 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.569522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.570586 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.570589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.571167 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.571168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.571232 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.571237 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.571305 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.571336 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260927 18:38:04.571424 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.568402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:36445 (local address 127.5.151.193:42115)
0927 18:38:04.568445 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.568447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.568570 (+ 123us) server_negotiation.cc:407] Connection header received
0927 18:38:04.568639 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.568640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.568648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.568704 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:04.568786 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.569436 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.570667 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.571048 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.571162 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.571172 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.571311 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.571348 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:38:04.571784 25006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8cb0f6b4684547a195470ae89b5faf" candidate_uuid: "60bc6f5b58224571aa258395cb6f1312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ca9801991645d28c70ab2c135f3be0" is_pre_election: true
I20260927 18:38:04.571930 25006 raft_consensus.cc:2454] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60bc6f5b58224571aa258395cb6f1312 in term 0.
I20260927 18:38:04.572079 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.568207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:33391 (local address 127.5.151.194:36643)
0927 18:38:04.568276 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.568289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.568338 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.568468 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.568469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.568472 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:04.568539 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.568540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.569409 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.569409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.570594 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.570595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.571802 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.571803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.571917 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.571920 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.571978 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.572016 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":57}
I20260927 18:38:04.572314 25073 leader_election.cc:302] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ca9801991645d28c70ab2c135f3be0, 60bc6f5b58224571aa258395cb6f1312; no voters:
I20260927 18:38:04.572360 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.568260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:36643 (local address 127.5.151.195:33391)
0927 18:38:04.568313 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.568315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.568323 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:38:04.568396 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.568397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.568406 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.568459 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:04.568614 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.569330 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.571215 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.571728 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.571782 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.571787 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.572245 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.572297 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:38:04.572420 25342 raft_consensus.cc:2781] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:04.572487 25342 raft_consensus.cc:482] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:04.572508 25342 raft_consensus.cc:3037] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:04.572567 25274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8cb0f6b4684547a195470ae89b5faf" candidate_uuid: "60bc6f5b58224571aa258395cb6f1312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" is_pre_election: true
I20260927 18:38:04.572681 25274 raft_consensus.cc:2454] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60bc6f5b58224571aa258395cb6f1312 in term 0.
I20260927 18:38:04.573169 25342 raft_consensus.cc:504] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.573288 25342 leader_election.cc:288] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [CANDIDATE]: Term 1 election: Requested vote from peers f97a4ea49b9f4feb9b37df44ea39b705 (127.5.151.195:33391), 21ca9801991645d28c70ab2c135f3be0 (127.5.151.193:42115)
I20260927 18:38:04.573438 25274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8cb0f6b4684547a195470ae89b5faf" candidate_uuid: "60bc6f5b58224571aa258395cb6f1312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f97a4ea49b9f4feb9b37df44ea39b705"
I20260927 18:38:04.573459 25006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8cb0f6b4684547a195470ae89b5faf" candidate_uuid: "60bc6f5b58224571aa258395cb6f1312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ca9801991645d28c70ab2c135f3be0"
I20260927 18:38:04.573511 25274 raft_consensus.cc:3037] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:04.573532 25006 raft_consensus.cc:3037] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:04.574170 25274 raft_consensus.cc:2454] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60bc6f5b58224571aa258395cb6f1312 in term 1.
I20260927 18:38:04.574175 25006 raft_consensus.cc:2454] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60bc6f5b58224571aa258395cb6f1312 in term 1.
I20260927 18:38:04.574344 25073 leader_election.cc:302] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ca9801991645d28c70ab2c135f3be0, 60bc6f5b58224571aa258395cb6f1312; no voters:
I20260927 18:38:04.574447 25342 raft_consensus.cc:2781] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:04.574587 25342 raft_consensus.cc:686] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 1 LEADER]: Becoming Leader. State: Replica: 60bc6f5b58224571aa258395cb6f1312, State: Running, Role: LEADER
I20260927 18:38:04.574667 25342 consensus_queue.cc:237] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } }
I20260927 18:38:04.575380 24861 catalog_manager.cc:5495] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60bc6f5b58224571aa258395cb6f1312 (127.5.151.194). New cstate: current_term: 1 leader_uuid: "60bc6f5b58224571aa258395cb6f1312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60bc6f5b58224571aa258395cb6f1312" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 43193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:04.583571 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.581672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.5.151.194:43193)
0927 18:38:04.581717 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.581718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.581725 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:38:04.581793 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.581793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.581807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.581864 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:04.581961 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.582354 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.583032 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.583196 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.583256 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.583364 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:04.583438 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:04.583485 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.583515 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.583529 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:38:04.584967 25274 raft_consensus.cc:1261] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 1 FOLLOWER]: Refusing update from remote peer 60bc6f5b58224571aa258395cb6f1312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:04.584945 25006 raft_consensus.cc:1261] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 1 FOLLOWER]: Refusing update from remote peer 60bc6f5b58224571aa258395cb6f1312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:04.585227 25342 consensus_queue.cc:1035] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97a4ea49b9f4feb9b37df44ea39b705" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:04.585374 25344 consensus_queue.cc:1035] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ca9801991645d28c70ab2c135f3be0" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 42115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:04.589555 24860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:04.589627 24860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:04.590291 24860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51fa0df916b64f57879610ca2c9b8205: Sending DeleteTablet for 3 replicas of tablet 3c8cb0f6b4684547a195470ae89b5faf
I20260927 18:38:04.590600 25120 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8cb0f6b4684547a195470ae89b5faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:04 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:43536
I20260927 18:38:04.590673 24986 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8cb0f6b4684547a195470ae89b5faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:04 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:53898
I20260927 18:38:04.590785 25254 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8cb0f6b4684547a195470ae89b5faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:04 UTC) from {username='kudu', principal='kudu/127.5.151.254@KRBTEST.COM'} at 127.0.0.1:37530
I20260927 18:38:04.590863 25357 tablet_replica.cc:330] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0: stopping tablet replica
I20260927 18:38:04.590952 25357 raft_consensus.cc:2229] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:04.591042 25357 raft_consensus.cc:2258] T 3c8cb0f6b4684547a195470ae89b5faf P 21ca9801991645d28c70ab2c135f3be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:04.591149 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 24923
I20260927 18:38:04.592747 25358 tablet_replica.cc:330] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: stopping tablet replica
I20260927 18:38:04.592842 25358 raft_consensus.cc:2229] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:04.592939 25358 raft_consensus.cc:2258] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:04.593091 25356 tablet_replica.cc:330] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: stopping tablet replica
I20260927 18:38:04.593175 25356 raft_consensus.cc:2229] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:04.593281 25356 raft_consensus.cc:2258] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:04.593684 25356 ts_tablet_manager.cc:1905] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:04.595261 25356 ts_tablet_manager.cc:1918] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:04.595330 25356 log.cc:1196] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/3c8cb0f6b4684547a195470ae89b5faf
I20260927 18:38:04.595618 25356 ts_tablet_manager.cc:1939] T 3c8cb0f6b4684547a195470ae89b5faf P 60bc6f5b58224571aa258395cb6f1312: Deleting consensus metadata
I20260927 18:38:04.596065 24847 catalog_manager.cc:4841] TS 60bc6f5b58224571aa258395cb6f1312 (127.5.151.194:43193): tablet 3c8cb0f6b4684547a195470ae89b5faf (table test-table [id=43f348748e034d61a732ca70b17885b9]) successfully deleted
W20260927 18:38:04.596738 24845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:42115 (error 108)
I20260927 18:38:04.597208 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.597083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:42115 (local address 127.0.0.1:53912)
0927 18:38:04.597142 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.597179 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:38:04.597270 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 25057
W20260927 18:38:04.598178 24845 catalog_manager.cc:4584] TS 21ca9801991645d28c70ab2c135f3be0 (127.5.151.193:42115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c8cb0f6b4684547a195470ae89b5faf: Network error: Client connection negotiation failed: client connection to 127.5.151.193:42115: connect: Connection refused (error 111)
I20260927 18:38:04.598953 25358 ts_tablet_manager.cc:1905] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:04.600209 25358 ts_tablet_manager.cc:1918] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:04.600291 25358 log.cc:1196] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/3c8cb0f6b4684547a195470ae89b5faf
I20260927 18:38:04.600517 25358 ts_tablet_manager.cc:1939] T 3c8cb0f6b4684547a195470ae89b5faf P f97a4ea49b9f4feb9b37df44ea39b705: Deleting consensus metadata
I20260927 18:38:04.600942 24845 catalog_manager.cc:4841] TS f97a4ea49b9f4feb9b37df44ea39b705 (127.5.151.195:33391): tablet 3c8cb0f6b4684547a195470ae89b5faf (table test-table [id=43f348748e034d61a732ca70b17885b9]) successfully deleted
I20260927 18:38:04.602352 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 25191
I20260927 18:38:04.607167 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 24829
2026-09-27T18:38:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1740 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T18:38:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:04Z Disabled control of system clock
I20260927 18:38:04.633040 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43265
--webserver_interface=127.5.151.254
--webserver_port=0
--builtin_ntp_servers=127.5.151.212:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:43265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:38:04.706352 25363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:04.706535 25363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:04.706566 25363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:04.707767 25363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:04.707808 25363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:04.707819 25363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:04.707873 25363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:04.707902 25363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:04.709215 25363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:43265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:04.709462 25363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:04.709759 25363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:04.714318 25369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.714327 25368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.714327 25371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.714502 25363 server_base.cc:1034] running on GCE node
I20260927 18:38:04.714745 25363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:04.715052 25363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:04.716209 25363 hybrid_clock.cc:648] HybridClock initialized: now 1790534284716198 us; error 42 us; skew 500 ppm
I20260927 18:38:04.717437 25363 webserver.cc:460] Webserver started at http://127.5.151.254:41601/ using document root <none> and password file <none>
I20260927 18:38:04.717674 25363 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:04.717751 25363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:04.717902 25363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:04.719039 25363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "e26095c86e8b4618b7119290a9ca4283"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "2486c9ed96557844b5f0d99af6debc46"
server_key_iv: "5b10387de11e5ba0e3f6cd44ed095f0a"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.719380 25363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "e26095c86e8b4618b7119290a9ca4283"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "2486c9ed96557844b5f0d99af6debc46"
server_key_iv: "5b10387de11e5ba0e3f6cd44ed095f0a"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.720611 25363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:04.721298 25377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.721463 25363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:04.721565 25363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "e26095c86e8b4618b7119290a9ca4283"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "2486c9ed96557844b5f0d99af6debc46"
server_key_iv: "5b10387de11e5ba0e3f6cd44ed095f0a"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.721639 25363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:04.737715 25363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:04.738045 25363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:04.738209 25363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:04.742238 25363 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:43265
I20260927 18:38:04.742277 25429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:43265 every 8 connection(s)
I20260927 18:38:04.742703 25363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:04.743705 25430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:04.746032 25430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283: Bootstrap starting.
I20260927 18:38:04.746697 25430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:04.746722 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 25363
I20260927 18:38:04.746873 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:04.747061 5727 external_mini_cluster.cc:1428] Setting key 0eace3c7bc7f526e9fdaf3b0dcf4966c
I20260927 18:38:04.747059 25430 log.cc:824] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:04.747997 25430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283: No bootstrap required, opened a new log
I20260927 18:38:04.750161 25430 raft_consensus.cc:348] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26095c86e8b4618b7119290a9ca4283" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43265 } }
I20260927 18:38:04.750322 25430 raft_consensus.cc:374] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:04.750353 25430 raft_consensus.cc:729] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26095c86e8b4618b7119290a9ca4283, State: Initialized, Role: FOLLOWER
I20260927 18:38:04.750459 25430 consensus_queue.cc:260] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26095c86e8b4618b7119290a9ca4283" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43265 } }
I20260927 18:38:04.750525 25430 raft_consensus.cc:388] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:04.750578 25430 raft_consensus.cc:482] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:04.750638 25430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:04.751216 25430 raft_consensus.cc:504] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26095c86e8b4618b7119290a9ca4283" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43265 } }
I20260927 18:38:04.751329 25430 leader_election.cc:302] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e26095c86e8b4618b7119290a9ca4283; no voters:
I20260927 18:38:04.751518 25430 leader_election.cc:288] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:04.751544 25435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:04.751675 25435 raft_consensus.cc:686] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [term 1 LEADER]: Becoming Leader. State: Replica: e26095c86e8b4618b7119290a9ca4283, State: Running, Role: LEADER
I20260927 18:38:04.751801 25430 sys_catalog.cc:564] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:04.751935 25435 consensus_queue.cc:237] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26095c86e8b4618b7119290a9ca4283" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43265 } }
I20260927 18:38:04.752257 25436 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e26095c86e8b4618b7119290a9ca4283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26095c86e8b4618b7119290a9ca4283" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43265 } } }
I20260927 18:38:04.752362 25436 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:04.752569 25443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:04.752738 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.747576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.5.151.254:43265)
0927 18:38:04.747754 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.747761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.747786 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:38:04.748290 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.748298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.748357 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.748471 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:04.748701 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.749533 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.750654 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.751084 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.751169 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.751878 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:04.751882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:04.751882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:04.752094 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:04.752133 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:04.752147 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:04.752207 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.752468 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.752546 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260927 18:38:04.753013 25443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:04.754492 25443 catalog_manager.cc:1347] Generated new cluster ID: 5366bb3f91324f50885d765a1d857ea9
I20260927 18:38:04.754542 25443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:04.753207 25437 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e26095c86e8b4618b7119290a9ca4283. Latest consensus state: current_term: 1 leader_uuid: "e26095c86e8b4618b7119290a9ca4283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26095c86e8b4618b7119290a9ca4283" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 43265 } } }
I20260927 18:38:04.757800 25437 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:04.773389 25443 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:04.773826 25443 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:04.780438 25443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283: Generated new TSK 0
I20260927 18:38:04.780583 25443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:04.785789 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.193:0
--local_ip_for_outbound_sockets=127.5.151.193
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43265
--builtin_ntp_servers=127.5.151.212:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:38:04.880741 25454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:04.880916 25454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:04.880936 25454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:04.882005 25454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:04.882042 25454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:04.882066 25454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.193
I20260927 18:38:04.883435 25454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.151.193
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.193
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:04.883661 25454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:04.883949 25454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:04.889369 25460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.889396 25459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:04.889393 25462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:04.889600 25454 server_base.cc:1034] running on GCE node
I20260927 18:38:04.889823 25454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:04.890055 25454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:04.891206 25454 hybrid_clock.cc:648] HybridClock initialized: now 1790534284891193 us; error 29 us; skew 500 ppm
I20260927 18:38:04.892351 25454 webserver.cc:460] Webserver started at http://127.5.151.193:42281/ using document root <none> and password file <none>
I20260927 18:38:04.892578 25454 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:04.892645 25454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:04.892799 25454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:04.893792 25454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data/instance:
uuid: "b762b8e636344ccaae6df113e111fa31"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "1f2e32cc41920387ae54cd85070a7433"
server_key_iv: "ac106408e08ccd2a2fccefef44c289e4"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.894143 25454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance:
uuid: "b762b8e636344ccaae6df113e111fa31"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "1f2e32cc41920387ae54cd85070a7433"
server_key_iv: "ac106408e08ccd2a2fccefef44c289e4"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.895462 25454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:04.896176 25468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.896343 25454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:04.896437 25454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/wal
uuid: "b762b8e636344ccaae6df113e111fa31"
format_stamp: "Formatted at 2026-09-27 18:38:04 on dist-test-slave-19nl"
server_key: "1f2e32cc41920387ae54cd85070a7433"
server_key_iv: "ac106408e08ccd2a2fccefef44c289e4"
server_key_version: "encryptionkey@0"
I20260927 18:38:04.896519 25454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:04.908378 25454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:04.908649 25454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:04.908797 25454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:04.909009 25454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:04.909296 25454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:04.909349 25454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.909389 25454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:04.909417 25454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:04.915053 25454 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.193:41661
I20260927 18:38:04.915107 25581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.193:41661 every 8 connection(s)
I20260927 18:38:04.915408 25454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:04.919929 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.916089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:42973 (local address 127.5.151.254:43265)
0927 18:38:04.916135 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:04.916138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:04.916390 (+ 252us) server_negotiation.cc:407] Connection header received
0927 18:38:04.916806 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:04.916807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:04.916817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:04.916866 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:04.917486 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.917967 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.918891 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.919060 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.919109 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.919718 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:04.919719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:04.919720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:04.919723 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:04.919734 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:04.919737 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:04.919777 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:04.919858 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 18:38:04.919876 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:38:04.920141 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:04.916088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43265 (local address 127.5.151.193:42973)
0927 18:38:04.916280 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:04.916310 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:04.916726 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:04.916911 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:04.916914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:04.916922 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:04.917420 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.917424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.918039 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.918040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.918824 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:04.918826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:04.919127 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:04.919128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:04.919222 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:04.919607 (+ 385us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:04.919607 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:04.919631 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:04.919650 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:04.919794 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:04.919798 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:04.919800 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:38:04.919848 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:38:04.919916 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260927 18:38:04.920632 25582 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43265
I20260927 18:38:04.920719 25582 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:04.920882 25582 heartbeater.cc:511] Master 127.5.151.254:43265 requested a full tablet report, sending...
I20260927 18:38:04.921000 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 25454
I20260927 18:38:04.921120 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:04.921320 5727 external_mini_cluster.cc:1428] Setting key 350418e66bb829ad847ee7af2d205e19
I20260927 18:38:04.921386 25394 ts_manager.cc:194] Registered new tserver with Master: b762b8e636344ccaae6df113e111fa31 (127.5.151.193:41661)
I20260927 18:38:04.921989 25394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.193:42973
I20260927 18:38:04.922428 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.194:0
--local_ip_for_outbound_sockets=127.5.151.194
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43265
--builtin_ntp_servers=127.5.151.212:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:38:05.000499 25585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:05.000684 25585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:05.000703 25585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:05.001708 25585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:05.001739 25585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:05.001762 25585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.194
I20260927 18:38:05.003000 25585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.151.194
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.194
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:05.003173 25585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:05.003391 25585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:05.007987 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:05.007999 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:05.007999 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:05.008095 25585 server_base.cc:1034] running on GCE node
I20260927 18:38:05.008347 25585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:05.008589 25585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:05.009740 25585 hybrid_clock.cc:648] HybridClock initialized: now 1790534285009726 us; error 46 us; skew 500 ppm
I20260927 18:38:05.010922 25585 webserver.cc:460] Webserver started at http://127.5.151.194:34961/ using document root <none> and password file <none>
I20260927 18:38:05.011148 25585 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:05.011217 25585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:05.011344 25585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:05.012312 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data/instance:
uuid: "45c77787696a4cdb999f57315d5588c1"
format_stamp: "Formatted at 2026-09-27 18:38:05 on dist-test-slave-19nl"
server_key: "3d46bb314dc2ec53e61a023cee40fc97"
server_key_iv: "a9aa98ce3c16da9907bbb0feeb61a78d"
server_key_version: "encryptionkey@0"
I20260927 18:38:05.012682 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance:
uuid: "45c77787696a4cdb999f57315d5588c1"
format_stamp: "Formatted at 2026-09-27 18:38:05 on dist-test-slave-19nl"
server_key: "3d46bb314dc2ec53e61a023cee40fc97"
server_key_iv: "a9aa98ce3c16da9907bbb0feeb61a78d"
server_key_version: "encryptionkey@0"
I20260927 18:38:05.013899 25585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:05.014544 25599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:05.014767 25585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:05.014859 25585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal
uuid: "45c77787696a4cdb999f57315d5588c1"
format_stamp: "Formatted at 2026-09-27 18:38:05 on dist-test-slave-19nl"
server_key: "3d46bb314dc2ec53e61a023cee40fc97"
server_key_iv: "a9aa98ce3c16da9907bbb0feeb61a78d"
server_key_version: "encryptionkey@0"
I20260927 18:38:05.014941 25585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:05.025839 25585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:05.026070 25585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:05.026202 25585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:05.026392 25585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:05.026736 25585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:05.026786 25585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:05.026815 25585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:05.026844 25585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:05.032255 25585 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.194:45035
I20260927 18:38:05.032292 25712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.194:45035 every 8 connection(s)
I20260927 18:38:05.032588 25585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:05.036803 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 25585
I20260927 18:38:05.036906 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:05.037124 5727 external_mini_cluster.cc:1428] Setting key 176c911b67e8c679cc302816c46ad6bd
I20260927 18:38:05.037307 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.033216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.194:34919 (local address 127.5.151.254:43265)
0927 18:38:05.033255 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.033257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.033439 (+ 182us) server_negotiation.cc:407] Connection header received
0927 18:38:05.033897 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.033898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.033910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.033969 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:05.034624 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.035109 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.036016 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.036228 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.036303 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.036989 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:05.036990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:05.036991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:05.036996 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:05.037013 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:05.037017 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:05.037075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.037193 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.037220 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:38:05.037534 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.033187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43265 (local address 127.5.151.194:34919)
0927 18:38:05.033339 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.033367 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:05.033799 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:05.034031 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:05.034034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:05.034043 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:05.034497 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.034500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.035206 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.035207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.035926 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.035928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.036309 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.036310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.036435 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.036838 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:05.036839 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:05.036876 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:05.036905 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:05.037090 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:05.037096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:05.037099 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:38:05.037175 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:38:05.037272 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260927 18:38:05.037997 25713 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43265
I20260927 18:38:05.038076 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.151.195:0
--local_ip_for_outbound_sockets=127.5.151.195
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43265
--builtin_ntp_servers=127.5.151.212:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:38:05.038089 25713 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:05.038254 25713 heartbeater.cc:511] Master 127.5.151.254:43265 requested a full tablet report, sending...
I20260927 18:38:05.038614 25394 ts_manager.cc:194] Registered new tserver with Master: 45c77787696a4cdb999f57315d5588c1 (127.5.151.194:45035)
I20260927 18:38:05.038975 25394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.194:34919
W20260927 18:38:05.112232 25716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:05.112397 25716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:05.112416 25716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:05.113536 25716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:05.113574 25716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:05.113618 25716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.151.195
I20260927 18:38:05.114881 25716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.151.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.151.195
--webserver_port=0
--tserver_master_addrs=127.5.151.254:43265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.151.195
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:05.115094 25716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:05.115348 25716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:05.119881 25721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:05.119972 25716 server_base.cc:1034] running on GCE node
W20260927 18:38:05.119915 25724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:05.119915 25722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:05.120306 25716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:05.120486 25716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:05.121614 25716 hybrid_clock.cc:648] HybridClock initialized: now 1790534285121602 us; error 26 us; skew 500 ppm
I20260927 18:38:05.122920 25716 webserver.cc:460] Webserver started at http://127.5.151.195:35613/ using document root <none> and password file <none>
I20260927 18:38:05.123137 25716 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:05.123181 25716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:05.123324 25716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:05.124331 25716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data/instance:
uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb"
format_stamp: "Formatted at 2026-09-27 18:38:05 on dist-test-slave-19nl"
server_key: "8d27cc1258e06ff35e592ca485350264"
server_key_iv: "3d3b388fe578824583c3c51098e82e60"
server_key_version: "encryptionkey@0"
I20260927 18:38:05.124679 25716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance:
uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb"
format_stamp: "Formatted at 2026-09-27 18:38:05 on dist-test-slave-19nl"
server_key: "8d27cc1258e06ff35e592ca485350264"
server_key_iv: "3d3b388fe578824583c3c51098e82e60"
server_key_version: "encryptionkey@0"
I20260927 18:38:05.125916 25716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:05.126598 25730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:05.126809 25716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:05.126893 25716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal
uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb"
format_stamp: "Formatted at 2026-09-27 18:38:05 on dist-test-slave-19nl"
server_key: "8d27cc1258e06ff35e592ca485350264"
server_key_iv: "3d3b388fe578824583c3c51098e82e60"
server_key_version: "encryptionkey@0"
I20260927 18:38:05.126956 25716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:05.143096 25716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:05.143316 25716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:05.143451 25716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:05.143646 25716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:05.143970 25716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:05.144002 25716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:05.144021 25716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:05.144065 25716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:05.149341 25716 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.195:43985
I20260927 18:38:05.149385 25843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.195:43985 every 8 connection(s)
I20260927 18:38:05.149646 25716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:05.152494 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 25716
I20260927 18:38:05.152602 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:05.152779 5727 external_mini_cluster.cc:1428] Setting key a70de63872ca45d97473068eaf1f284e
I20260927 18:38:05.154608 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.150227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.195:40437 (local address 127.5.151.254:43265)
0927 18:38:05.150294 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.150296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.150498 (+ 202us) server_negotiation.cc:407] Connection header received
0927 18:38:05.150991 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.150992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.151006 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.151055 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:05.151662 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.152211 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.153349 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.153588 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.153659 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.154320 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:05.154322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:05.154322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:05.154326 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:05.154340 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:05.154343 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:05.154410 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.154486 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.154511 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:38:05.154801 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.150241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:43265 (local address 127.5.151.195:40437)
0927 18:38:05.150394 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.150422 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:05.150887 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:05.151124 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:05.151128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:05.151144 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:05.151572 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.151576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.152306 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.152307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.153264 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.153272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.153675 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.153676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.153802 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.154181 (+ 379us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:05.154181 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:05.154204 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:05.154223 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:05.154414 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:05.154419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:05.154421 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:38:05.154488 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:38:05.154606 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260927 18:38:05.155190 25844 heartbeater.cc:347] Connected to a master server at 127.5.151.254:43265
I20260927 18:38:05.155273 25844 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:05.155444 25844 heartbeater.cc:511] Master 127.5.151.254:43265 requested a full tablet report, sending...
I20260927 18:38:05.155747 25394 ts_manager.cc:194] Registered new tserver with Master: 382180b7d8db43cdb69cbcc2e3a4b5eb (127.5.151.195:43985)
I20260927 18:38:05.156087 25394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.151.195:40437
I20260927 18:38:05.164425 5727 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:05.168515 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.165725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.5.151.254:43265)
0927 18:38:05.165799 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.165801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.165892 (+ 91us) server_negotiation.cc:407] Connection header received
0927 18:38:05.165968 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.165969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.165985 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.166044 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:05.166218 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.166717 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.167460 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.167635 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.167706 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.168261 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:05.168263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:05.168263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:05.168267 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:05.168280 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:05.168284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:05.168357 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.168417 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.168436 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:38:05.170063 25394 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:05.171584 25394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:05.177258 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.173565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.5.151.194:45035)
0927 18:38:05.173752 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.173757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.173783 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:38:05.173819 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.173821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.173834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.173907 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:05.174091 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.174979 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.176264 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.176896 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.176981 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.177066 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.177142 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.177178 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260927 18:38:05.177248 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.173485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:45035 (local address 127.0.0.1:56648)
0927 18:38:05.173630 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.173675 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:05.173738 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:05.173933 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:05.173937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:05.173940 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:05.174005 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.174006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.175060 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.175061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.176065 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.176068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.176981 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.176982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.177054 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.177065 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:38:05.177132 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:38:05.177179 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260927 18:38:05.177556 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.173510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:41661 (local address 127.0.0.1:54148)
0927 18:38:05.173918 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.173928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:05.173980 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:05.174329 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:05.174330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:05.174332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:05.174364 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.174366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.175596 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.175597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.176395 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.176397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.177320 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.177321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.177398 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.177403 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:38:05.177455 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 18:38:05.177488 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":95,"threads_started":1}
I20260927 18:38:05.177788 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.173600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.5.151.193:41661)
0927 18:38:05.173787 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.173791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.174044 (+ 253us) server_negotiation.cc:407] Connection header received
0927 18:38:05.174090 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.174092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.174107 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.174184 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:05.174444 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.175514 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.176465 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.177135 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.177205 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.177282 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.177658 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.177708 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260927 18:38:05.177978 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.173705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:43985 (local address 127.0.0.1:48204)
0927 18:38:05.173994 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.174004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:05.174052 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:05.174250 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:05.174251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:05.174253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:05.174306 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.174307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.175564 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.175565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.176573 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.176575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.177763 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.177764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.177823 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.177828 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:38:05.177880 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 18:38:05.177920 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":58,"threads_started":1}
I20260927 18:38:05.177977 25647 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fd26897c04ff0a13f5c1afa90d88e (DEFAULT_TABLE table=test-table [id=31b3772095ac44dfbc3034f20fdad3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:05.178170 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.173886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.5.151.195:43985)
0927 18:38:05.174062 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.174066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.174075 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:05.174112 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.174114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.174126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.174198 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:05.174380 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.175474 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.176652 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.177593 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.177654 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.177732 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.178062 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.178109 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260927 18:38:05.178360 25647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2fd26897c04ff0a13f5c1afa90d88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:05.178406 25516 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fd26897c04ff0a13f5c1afa90d88e (DEFAULT_TABLE table=test-table [id=31b3772095ac44dfbc3034f20fdad3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:05.178740 25516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2fd26897c04ff0a13f5c1afa90d88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:05.181648 25858 tablet_bootstrap.cc:492] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Bootstrap starting.
I20260927 18:38:05.181766 25859 tablet_bootstrap.cc:492] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31: Bootstrap starting.
I20260927 18:38:05.182405 25859 tablet_bootstrap.cc:654] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:05.182368 25778 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fd26897c04ff0a13f5c1afa90d88e (DEFAULT_TABLE table=test-table [id=31b3772095ac44dfbc3034f20fdad3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:05.182657 25778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2fd26897c04ff0a13f5c1afa90d88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:05.182744 25859 log.cc:824] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:05.183408 25859 tablet_bootstrap.cc:492] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31: No bootstrap required, opened a new log
I20260927 18:38:05.183487 25859 ts_tablet_manager.cc:1397] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:38:05.185078 25861 tablet_bootstrap.cc:492] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Bootstrap starting.
I20260927 18:38:05.185261 25858 tablet_bootstrap.cc:654] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:05.185577 25858 log.cc:824] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:05.185671 25861 tablet_bootstrap.cc:654] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:05.185580 25859 raft_consensus.cc:348] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.185747 25859 raft_consensus.cc:374] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:05.185789 25859 raft_consensus.cc:729] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b762b8e636344ccaae6df113e111fa31, State: Initialized, Role: FOLLOWER
I20260927 18:38:05.185899 25859 consensus_queue.cc:260] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.185963 25861 log.cc:824] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:05.186169 25859 ts_tablet_manager.cc:1428] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:38:05.186214 25582 heartbeater.cc:503] Master 127.5.151.254:43265 was elected leader, sending a full tablet report...
I20260927 18:38:05.186267 25858 tablet_bootstrap.cc:492] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: No bootstrap required, opened a new log
I20260927 18:38:05.186343 25858 ts_tablet_manager.cc:1397] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 18:38:05.187371 25861 tablet_bootstrap.cc:492] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: No bootstrap required, opened a new log
I20260927 18:38:05.187479 25861 ts_tablet_manager.cc:1397] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:38:05.188907 25858 raft_consensus.cc:348] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.189075 25858 raft_consensus.cc:374] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:05.189117 25858 raft_consensus.cc:729] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c77787696a4cdb999f57315d5588c1, State: Initialized, Role: FOLLOWER
I20260927 18:38:05.189231 25858 consensus_queue.cc:260] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.189467 25858 ts_tablet_manager.cc:1428] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:05.189531 25713 heartbeater.cc:503] Master 127.5.151.254:43265 was elected leader, sending a full tablet report...
I20260927 18:38:05.189944 25861 raft_consensus.cc:348] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.190086 25861 raft_consensus.cc:374] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:05.190135 25861 raft_consensus.cc:729] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382180b7d8db43cdb69cbcc2e3a4b5eb, State: Initialized, Role: FOLLOWER
I20260927 18:38:05.190246 25861 consensus_queue.cc:260] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.190498 25861 ts_tablet_manager.cc:1428] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:05.190524 25844 heartbeater.cc:503] Master 127.5.151.254:43265 was elected leader, sending a full tablet report...
W20260927 18:38:05.283128 25714 tablet.cc:2369] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:05.300491 25863 raft_consensus.cc:482] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:05.300589 25863 raft_consensus.cc:504] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.300891 25863 leader_election.cc:288] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45c77787696a4cdb999f57315d5588c1 (127.5.151.194:45035), 382180b7d8db43cdb69cbcc2e3a4b5eb (127.5.151.195:43985)
I20260927 18:38:05.303692 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.301124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:46367 (local address 127.5.151.194:45035)
0927 18:38:05.301192 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.301195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.301341 (+ 146us) server_negotiation.cc:407] Connection header received
0927 18:38:05.301380 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.301381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.301391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.301450 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:05.301618 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.302069 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.302985 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.303349 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.303404 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.303410 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.303577 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.303621 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:38:05.303725 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.301054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.195:43985 (local address 127.5.151.193:56443)
0927 18:38:05.301108 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.301120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:05.301166 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:05.301349 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:05.301351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:05.301352 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:05.301429 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.301432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.302085 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.302087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.302919 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.302921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.303467 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.303468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.303524 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.303527 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:38:05.303595 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:38:05.303633 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":27}
I20260927 18:38:05.303730 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.301159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.151.193:56443 (local address 127.5.151.195:43985)
0927 18:38:05.301197 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.301199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.301206 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:38:05.301236 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.301237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.301246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.301320 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:05.301506 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.302006 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.303033 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.303361 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.303448 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.303456 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.303617 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.303656 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:38:05.303937 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.301056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.194:45035 (local address 127.5.151.193:46367)
0927 18:38:05.301269 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.301279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:05.301335 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:05.301479 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:05.301480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:05.301482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:05.301553 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.301554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.302152 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.302153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.302907 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:05.302909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.303433 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.303434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:05.303511 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.303514 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:38:05.303554 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 18:38:05.303609 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":12,"thread_start_us":123,"threads_started":1}
I20260927 18:38:05.304057 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fd26897c04ff0a13f5c1afa90d88e" candidate_uuid: "b762b8e636344ccaae6df113e111fa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" is_pre_election: true
I20260927 18:38:05.304210 25798 raft_consensus.cc:2454] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b762b8e636344ccaae6df113e111fa31 in term 0.
I20260927 18:38:05.304224 25667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fd26897c04ff0a13f5c1afa90d88e" candidate_uuid: "b762b8e636344ccaae6df113e111fa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c77787696a4cdb999f57315d5588c1" is_pre_election: true
I20260927 18:38:05.304350 25667 raft_consensus.cc:2454] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b762b8e636344ccaae6df113e111fa31 in term 0.
I20260927 18:38:05.304488 25471 leader_election.cc:302] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382180b7d8db43cdb69cbcc2e3a4b5eb, b762b8e636344ccaae6df113e111fa31; no voters:
I20260927 18:38:05.304606 25863 raft_consensus.cc:2781] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:05.304672 25863 raft_consensus.cc:482] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:05.304689 25863 raft_consensus.cc:3037] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:05.305308 25863 raft_consensus.cc:504] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.305440 25863 leader_election.cc:288] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [CANDIDATE]: Term 1 election: Requested vote from peers 45c77787696a4cdb999f57315d5588c1 (127.5.151.194:45035), 382180b7d8db43cdb69cbcc2e3a4b5eb (127.5.151.195:43985)
I20260927 18:38:05.305567 25667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fd26897c04ff0a13f5c1afa90d88e" candidate_uuid: "b762b8e636344ccaae6df113e111fa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c77787696a4cdb999f57315d5588c1"
I20260927 18:38:05.305586 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fd26897c04ff0a13f5c1afa90d88e" candidate_uuid: "b762b8e636344ccaae6df113e111fa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb"
I20260927 18:38:05.305641 25798 raft_consensus.cc:3037] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:05.305660 25667 raft_consensus.cc:3037] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:05.306332 25798 raft_consensus.cc:2454] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b762b8e636344ccaae6df113e111fa31 in term 1.
I20260927 18:38:05.306344 25667 raft_consensus.cc:2454] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b762b8e636344ccaae6df113e111fa31 in term 1.
I20260927 18:38:05.306488 25471 leader_election.cc:302] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382180b7d8db43cdb69cbcc2e3a4b5eb, b762b8e636344ccaae6df113e111fa31; no voters:
I20260927 18:38:05.306620 25863 raft_consensus.cc:2781] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:05.306771 25863 raft_consensus.cc:686] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 1 LEADER]: Becoming Leader. State: Replica: b762b8e636344ccaae6df113e111fa31, State: Running, Role: LEADER
I20260927 18:38:05.306870 25863 consensus_queue.cc:237] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } }
I20260927 18:38:05.307628 25394 catalog_manager.cc:5495] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 reported cstate change: term changed from 0 to 1, leader changed from <none> to b762b8e636344ccaae6df113e111fa31 (127.5.151.193). New cstate: current_term: 1 leader_uuid: "b762b8e636344ccaae6df113e111fa31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b762b8e636344ccaae6df113e111fa31" member_type: VOTER last_known_addr { host: "127.5.151.193" port: 41661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:05.320315 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.318295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.5.151.193:41661)
0927 18:38:05.318355 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:05.318356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:05.318364 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:38:05.318405 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:05.318406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:05.318420 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:05.318487 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:05.318629 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.319036 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.319691 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:05.319859 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:05.319905 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:05.320029 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:05.320110 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:05.320166 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:05.320200 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 18:38:05.320215 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:38:05.321906 25667 raft_consensus.cc:1261] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 1 FOLLOWER]: Refusing update from remote peer b762b8e636344ccaae6df113e111fa31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:05.321918 25798 raft_consensus.cc:1261] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 1 FOLLOWER]: Refusing update from remote peer b762b8e636344ccaae6df113e111fa31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:05.322252 25868 consensus_queue.cc:1035] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c77787696a4cdb999f57315d5588c1" member_type: VOTER last_known_addr { host: "127.5.151.194" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:05.322396 25863 consensus_queue.cc:1035] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382180b7d8db43cdb69cbcc2e3a4b5eb" member_type: VOTER last_known_addr { host: "127.5.151.195" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:05.324481 25871 mvcc.cc:205] Tried to move back new op lower bound from 7334028432675979264 to 7334028432617451520. Current Snapshot: MvccSnapshot[applied={T|T < 7334028432675979264}]
I20260927 18:38:05.324753 25873 mvcc.cc:205] Tried to move back new op lower bound from 7334028432675979264 to 7334028432617451520. Current Snapshot: MvccSnapshot[applied={T|T < 7334028432675979264}]
I20260927 18:38:05.325332 25872 mvcc.cc:205] Tried to move back new op lower bound from 7334028432675979264 to 7334028432617451520. Current Snapshot: MvccSnapshot[applied={T|T < 7334028432675979264}]
I20260927 18:38:05.326736 25394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:05.326848 25394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:05.327555 25394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e26095c86e8b4618b7119290a9ca4283: Sending DeleteTablet for 3 replicas of tablet 1d2fd26897c04ff0a13f5c1afa90d88e
I20260927 18:38:05.327832 25516 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fd26897c04ff0a13f5c1afa90d88e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:05 UTC) from {username='slave'} at 127.0.0.1:54148
I20260927 18:38:05.327883 25778 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fd26897c04ff0a13f5c1afa90d88e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:05 UTC) from {username='slave'} at 127.0.0.1:48204
I20260927 18:38:05.328051 25880 tablet_replica.cc:330] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31: stopping tablet replica
I20260927 18:38:05.328083 25647 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fd26897c04ff0a13f5c1afa90d88e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:05 UTC) from {username='slave'} at 127.0.0.1:56648
I20260927 18:38:05.328150 25880 raft_consensus.cc:2229] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:05.328256 25880 raft_consensus.cc:2258] T 1d2fd26897c04ff0a13f5c1afa90d88e P b762b8e636344ccaae6df113e111fa31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:05.328401 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 25454
I20260927 18:38:05.329923 25881 tablet_replica.cc:330] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: stopping tablet replica
I20260927 18:38:05.330018 25881 raft_consensus.cc:2229] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:05.330114 25881 raft_consensus.cc:2258] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:05.330251 25882 tablet_replica.cc:330] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: stopping tablet replica
I20260927 18:38:05.330335 25882 raft_consensus.cc:2229] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:05.330425 25882 raft_consensus.cc:2258] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:05.330811 25882 ts_tablet_manager.cc:1905] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:05.331182 25881 ts_tablet_manager.cc:1905] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:05.332610 25881 ts_tablet_manager.cc:1918] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:05.332679 25882 ts_tablet_manager.cc:1918] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:05.332700 25881 log.cc:1196] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-2/wal/wals/1d2fd26897c04ff0a13f5c1afa90d88e
I20260927 18:38:05.332747 25882 log.cc:1196] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Deleting WAL directory at /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534055081918-5727-0/minicluster-data/ts-1/wal/wals/1d2fd26897c04ff0a13f5c1afa90d88e
I20260927 18:38:05.332953 25881 ts_tablet_manager.cc:1939] T 1d2fd26897c04ff0a13f5c1afa90d88e P 382180b7d8db43cdb69cbcc2e3a4b5eb: Deleting consensus metadata
I20260927 18:38:05.333014 25882 ts_tablet_manager.cc:1939] T 1d2fd26897c04ff0a13f5c1afa90d88e P 45c77787696a4cdb999f57315d5588c1: Deleting consensus metadata
I20260927 18:38:05.333380 25380 catalog_manager.cc:4841] TS 382180b7d8db43cdb69cbcc2e3a4b5eb (127.5.151.195:43985): tablet 1d2fd26897c04ff0a13f5c1afa90d88e (table test-table [id=31b3772095ac44dfbc3034f20fdad3ed]) successfully deleted
I20260927 18:38:05.333391 25378 catalog_manager.cc:4841] TS 45c77787696a4cdb999f57315d5588c1 (127.5.151.194:45035): tablet 1d2fd26897c04ff0a13f5c1afa90d88e (table test-table [id=31b3772095ac44dfbc3034f20fdad3ed]) successfully deleted
W20260927 18:38:05.334923 25380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.151.193:41661 (error 108)
I20260927 18:38:05.335182 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 25585
I20260927 18:38:05.335790 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:05.335260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.193:41661 (local address 127.0.0.1:54172)
0927 18:38:05.335717 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:05.335759 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.193:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
W20260927 18:38:05.335911 25380 catalog_manager.cc:4584] TS b762b8e636344ccaae6df113e111fa31 (127.5.151.193:41661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d2fd26897c04ff0a13f5c1afa90d88e: Network error: Client connection negotiation failed: client connection to 127.5.151.193:41661: connect: Connection refused (error 111)
I20260927 18:38:05.340023 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 25716
I20260927 18:38:05.344300 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 25363
2026-09-27T18:38:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (735 ms)
[----------] 6 tests from EncryptionPolicyTest (8215 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:05 dist-test-slave-19nl krb5kdc[25886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:05 dist-test-slave-19nl krb5kdc[25886](info): set up 2 sockets
Sep 27 18:38:05 dist-test-slave-19nl krb5kdc[25886](info): commencing operation
krb5kdc: starting...
W20260927 18:38:06.334372 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:06 dist-test-slave-19nl krb5kdc[25886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:06.442363 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:06.518360 25899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:06.518566 25899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:06.518615 25899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:06.519701 25899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:06.519763 25899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:06.519788 25899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:06.519819 25899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:06.519841 25899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:06.521178 25899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:06.521428 25899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:06.521631 25899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:06.526355 25907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:06.526322 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:06.526404 25905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:06.526410 25899 server_base.cc:1034] running on GCE node
I20260927 18:38:06.526738 25899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:06.527022 25899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:06.528168 25899 hybrid_clock.cc:648] HybridClock initialized: now 1790534286528155 us; error 25 us; skew 500 ppm
Sep 27 18:38:06 dist-test-slave-19nl krb5kdc[25886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:06.530467 25899 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:06.530885 25899 webserver.cc:460] Webserver started at http://127.0.0.1:37293/ using document root <none> and password file <none>
I20260927 18:38:06.531098 25899 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:06.531141 25899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:06.531299 25899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:06.532291 25899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "7b2c18a88a3748c6b4939207505f1810"
format_stamp: "Formatted at 2026-09-27 18:38:06 on dist-test-slave-19nl"
server_key: "073bda9d22f3b75f9e7777899da89e7c"
server_key_iv: "12a4cac21362bcb1b9e0503162b7fe5f"
server_key_version: "encryptionkey@0"
I20260927 18:38:06.532622 25899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "7b2c18a88a3748c6b4939207505f1810"
format_stamp: "Formatted at 2026-09-27 18:38:06 on dist-test-slave-19nl"
server_key: "073bda9d22f3b75f9e7777899da89e7c"
server_key_iv: "12a4cac21362bcb1b9e0503162b7fe5f"
server_key_version: "encryptionkey@0"
I20260927 18:38:06.533836 25899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:06.534596 25914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:06.534775 25899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:38:06.534868 25899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "7b2c18a88a3748c6b4939207505f1810"
format_stamp: "Formatted at 2026-09-27 18:38:06 on dist-test-slave-19nl"
server_key: "073bda9d22f3b75f9e7777899da89e7c"
server_key_iv: "12a4cac21362bcb1b9e0503162b7fe5f"
server_key_version: "encryptionkey@0"
I20260927 18:38:06.534946 25899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:06.543414 25899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:06.545821 25899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:06.545962 25899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:06.549683 25899 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46385
I20260927 18:38:06.549710 25966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46385 every 8 connection(s)
I20260927 18:38:06.550014 25899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:06.550877 25967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:06.552861 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810: Bootstrap starting.
I20260927 18:38:06.553359 25967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:06.553575 25967 log.cc:824] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:06.554169 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810: No bootstrap required, opened a new log
I20260927 18:38:06.555657 25967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2c18a88a3748c6b4939207505f1810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260927 18:38:06.555779 25967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:06.555799 25967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2c18a88a3748c6b4939207505f1810, State: Initialized, Role: FOLLOWER
I20260927 18:38:06.555920 25967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2c18a88a3748c6b4939207505f1810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260927 18:38:06.555992 25967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:06.556013 25967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:06.556039 25967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:06.556547 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 25899
I20260927 18:38:06.556552 25967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2c18a88a3748c6b4939207505f1810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260927 18:38:06.556658 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:06.556645 25967 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2c18a88a3748c6b4939207505f1810; no voters:
I20260927 18:38:06.556794 25967 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:06.556841 25970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:06.556860 5727 external_mini_cluster.cc:1428] Setting key 2d11f0b708d99d75b45d5da3b782b456
I20260927 18:38:06.556980 25970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [term 1 LEADER]: Becoming Leader. State: Replica: 7b2c18a88a3748c6b4939207505f1810, State: Running, Role: LEADER
I20260927 18:38:06.557020 25967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:06.557056 25970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2c18a88a3748c6b4939207505f1810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } }
I20260927 18:38:06.557359 25970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2c18a88a3748c6b4939207505f1810. Latest consensus state: current_term: 1 leader_uuid: "7b2c18a88a3748c6b4939207505f1810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2c18a88a3748c6b4939207505f1810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } } }
I20260927 18:38:06.557437 25970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:06.557579 25971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2c18a88a3748c6b4939207505f1810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2c18a88a3748c6b4939207505f1810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46385 } } }
I20260927 18:38:06.557708 25971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:06.558288 25977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:06.558789 25977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:06.560192 25977 catalog_manager.cc:1347] Generated new cluster ID: 87b744318fb04068b9dc5c7d04691bb4
I20260927 18:38:06.560242 25977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:38:06 dist-test-slave-19nl krb5kdc[25886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:38:06.567130 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:06.557426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59799 (local address 127.0.0.1:46385)
0927 18:38:06.557622 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:06.557632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:06.557901 (+ 269us) server_negotiation.cc:407] Connection header received
0927 18:38:06.558110 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:06.558117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:06.558175 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:06.558306 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:06.559201 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:06.560434 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:06.561181 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:06.561603 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:06.561717 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:06.564065 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:06.564075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:06.564076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:06.564086 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:06.565521 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:06.565934 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:06.565937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:06.565937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:06.566036 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:06.566264 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:06.566265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:06.566266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:06.566485 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:06.566666 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:06.566804 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:38:06.566910 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":61,"threads_started":1}
I20260927 18:38:06.567988 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:06.573272 25977 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:06.573942 25977 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:06.575336 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:06.569186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41945 (local address 127.0.0.1:46385)
0927 18:38:06.569216 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:06.569218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:06.569399 (+ 181us) server_negotiation.cc:407] Connection header received
0927 18:38:06.569460 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:06.569461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:06.569469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:06.569508 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:06.569981 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:06.570449 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:06.571176 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:06.571404 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:06.571475 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:06.572573 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:06.572574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:06.572575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:06.572581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:06.574157 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:06.574509 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:06.574510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:06.574510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:06.574582 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:06.574799 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:06.574800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:06.574801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:06.574907 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:06.575034 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:06.575153 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:38:06.575209 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 18:38:06.580096 25977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2c18a88a3748c6b4939207505f1810: Generated new TSK 0
I20260927 18:38:06.580246 25977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:06.582896 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 25899
W20260927 18:38:06.586431 25884 connection.cc:535] client connection to 127.0.0.1:46385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46385): Connection reset by peer (error 104)
2026-09-27T18:38:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:06 dist-test-slave-19nl krb5kdc[25998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:06 dist-test-slave-19nl krb5kdc[25998](info): set up 2 sockets
Sep 27 18:38:06 dist-test-slave-19nl krb5kdc[25998](info): commencing operation
krb5kdc: starting...
W20260927 18:38:07.570048 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:07 dist-test-slave-19nl krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:07.674878 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:07.750449 26011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:07.750676 26011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:07.750715 26011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:07.751783 26011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:07.751842 26011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:07.751881 26011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:07.751902 26011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:07.751930 26011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:07.753347 26011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:07.753607 26011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:07.753815 26011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:07.758128 26017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:07.758108 26019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:07.758114 26016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:07.758303 26011 server_base.cc:1034] running on GCE node
I20260927 18:38:07.758493 26011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:07.758816 26011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:07.759964 26011 hybrid_clock.cc:648] HybridClock initialized: now 1790534287759957 us; error 39 us; skew 500 ppm
Sep 27 18:38:07 dist-test-slave-19nl krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:07.765218 26011 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:07.765630 26011 webserver.cc:460] Webserver started at http://127.0.0.1:37159/ using document root <none> and password file <none>
I20260927 18:38:07.765838 26011 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:07.765899 26011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:07.766033 26011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:07.767002 26011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "63f8c840d36b465da2f27eba0637f4a0"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-19nl"
server_key: "12b223c804461621cfa347272b8a0531"
server_key_iv: "fdaebc9a1e2d9bac62d9e264484c187c"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.767359 26011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "63f8c840d36b465da2f27eba0637f4a0"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-19nl"
server_key: "12b223c804461621cfa347272b8a0531"
server_key_iv: "fdaebc9a1e2d9bac62d9e264484c187c"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.768548 26011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:07.769174 26026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:07.769325 26011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:07.769410 26011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "63f8c840d36b465da2f27eba0637f4a0"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-19nl"
server_key: "12b223c804461621cfa347272b8a0531"
server_key_iv: "fdaebc9a1e2d9bac62d9e264484c187c"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.769483 26011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:07.793812 26011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:07.795907 26011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:07.796063 26011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:07.799849 26011 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42321
I20260927 18:38:07.799882 26078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42321 every 8 connection(s)
I20260927 18:38:07.800190 26011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:07.801002 26079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:07.803000 26079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0: Bootstrap starting.
I20260927 18:38:07.803512 26079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:07.803812 26079 log.cc:824] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:07.804455 26079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0: No bootstrap required, opened a new log
I20260927 18:38:07.805962 26079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f8c840d36b465da2f27eba0637f4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260927 18:38:07.806107 26079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:07.806129 26079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f8c840d36b465da2f27eba0637f4a0, State: Initialized, Role: FOLLOWER
I20260927 18:38:07.806294 26079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f8c840d36b465da2f27eba0637f4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260927 18:38:07.806354 26079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:07.806375 26079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:07.806438 26079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:07.807010 26079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f8c840d36b465da2f27eba0637f4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260927 18:38:07.807121 26079 leader_election.cc:302] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63f8c840d36b465da2f27eba0637f4a0; no voters:
I20260927 18:38:07.807265 26079 leader_election.cc:288] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:07.807291 26082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:07.807492 26082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [term 1 LEADER]: Becoming Leader. State: Replica: 63f8c840d36b465da2f27eba0637f4a0, State: Running, Role: LEADER
I20260927 18:38:07.807499 26079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:07.807587 26082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f8c840d36b465da2f27eba0637f4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260927 18:38:07.808099 26083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63f8c840d36b465da2f27eba0637f4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f8c840d36b465da2f27eba0637f4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } } }
I20260927 18:38:07.808326 26083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:07.808243 26084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63f8c840d36b465da2f27eba0637f4a0. Latest consensus state: current_term: 1 leader_uuid: "63f8c840d36b465da2f27eba0637f4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f8c840d36b465da2f27eba0637f4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } } }
I20260927 18:38:07.808518 26084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:07.808552 26092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:07.808631 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26011
I20260927 18:38:07.808745 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:07.808943 5727 external_mini_cluster.cc:1428] Setting key 389809e22e6c3c0be5896d0d01a02f1b
I20260927 18:38:07.809450 26092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:07.810927 26092 catalog_manager.cc:1347] Generated new cluster ID: 4df1475a40614b19af8542e2c6de62a0
I20260927 18:38:07.810974 26092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:07.814888 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:07.809688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35011 (local address 127.0.0.1:42321)
0927 18:38:07.809878 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:07.809886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:07.809917 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:38:07.810018 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:07.810024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:07.810091 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:07.810240 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:07.810435 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.811804 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.812829 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.813451 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.813561 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:07.814200 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:07.814205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:07.814205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:07.814412 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:07.814449 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:07.814465 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:07.814574 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:07.814669 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 18:38:07.814694 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260927 18:38:07.816002 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:07.820722 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:07.817470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34001 (local address 127.0.0.1:42321)
0927 18:38:07.817507 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:07.817509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:07.817668 (+ 159us) server_negotiation.cc:407] Connection header received
0927 18:38:07.817694 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:07.817695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:07.817704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:07.817770 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:07.817864 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.818595 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.819522 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.819843 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.819916 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:07.820498 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:07.820499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:07.820499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:07.820503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:07.820516 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:07.820520 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:07.820580 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:07.820649 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:38:07.820659 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:38:07.827632 26092 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:07.828063 26092 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:07.834028 26092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63f8c840d36b465da2f27eba0637f4a0: Generated new TSK 0
I20260927 18:38:07.834201 26092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:07.836781 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26011
W20260927 18:38:07.840662 25996 connection.cc:535] client connection to 127.0.0.1:42321 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42321): Connection reset by peer (error 104)
2026-09-27T18:38:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:07 dist-test-slave-19nl krb5kdc[26110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:07 dist-test-slave-19nl krb5kdc[26110](info): set up 2 sockets
Sep 27 18:38:07 dist-test-slave-19nl krb5kdc[26110](info): commencing operation
krb5kdc: starting...
W20260927 18:38:08.818348 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:08 dist-test-slave-19nl krb5kdc[26110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:08.922742 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:08.999390 26123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:08.999576 26123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:08.999598 26123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:09.000711 26123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:09.000751 26123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:09.000764 26123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:09.000777 26123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:09.000823 26123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:09.002218 26123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:09.002453 26123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:09.002702 26123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:09.007197 26128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:09.007230 26129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:09.007264 26123 server_base.cc:1034] running on GCE node
W20260927 18:38:09.007220 26131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:09.007643 26123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:09.007952 26123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:09.009114 26123 hybrid_clock.cc:648] HybridClock initialized: now 1790534289009110 us; error 35 us; skew 500 ppm
Sep 27 18:38:09 dist-test-slave-19nl krb5kdc[26110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:09.011565 26123 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:09.011957 26123 webserver.cc:460] Webserver started at http://127.0.0.1:44787/ using document root <none> and password file <none>
I20260927 18:38:09.012194 26123 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:09.012262 26123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:09.012399 26123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:09.013401 26123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "e851855212c54d2db4c8cb3a7419340a"
format_stamp: "Formatted at 2026-09-27 18:38:09 on dist-test-slave-19nl"
server_key: "62bd0ea6cc72e6736513c80e3ae49c9e"
server_key_iv: "eb2ced275f0ca94b0f6036082b039fd5"
server_key_version: "encryptionkey@0"
I20260927 18:38:09.013726 26123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "e851855212c54d2db4c8cb3a7419340a"
format_stamp: "Formatted at 2026-09-27 18:38:09 on dist-test-slave-19nl"
server_key: "62bd0ea6cc72e6736513c80e3ae49c9e"
server_key_iv: "eb2ced275f0ca94b0f6036082b039fd5"
server_key_version: "encryptionkey@0"
I20260927 18:38:09.015049 26123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:09.015743 26138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:09.015898 26123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:09.015996 26123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "e851855212c54d2db4c8cb3a7419340a"
format_stamp: "Formatted at 2026-09-27 18:38:09 on dist-test-slave-19nl"
server_key: "62bd0ea6cc72e6736513c80e3ae49c9e"
server_key_iv: "eb2ced275f0ca94b0f6036082b039fd5"
server_key_version: "encryptionkey@0"
I20260927 18:38:09.016064 26123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:09.019117 26123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:09.021185 26123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:09.021327 26123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:09.025118 26123 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43755
I20260927 18:38:09.025147 26190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43755 every 8 connection(s)
I20260927 18:38:09.025444 26123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:09.026358 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:09.026687 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26123
I20260927 18:38:09.026808 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:09.026994 5727 external_mini_cluster.cc:1428] Setting key 4897248ce658cc594f39e22410ceb6b4
I20260927 18:38:09.029172 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:09.027460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41355 (local address 127.0.0.1:43755)
0927 18:38:09.027620 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:09.027628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:09.027658 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:38:09.028047 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:09.028052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:09.028074 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:09.028185 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:09.028743 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:09.028746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:09.028746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:09.028750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:09.028782 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:09.028792 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:09.028843 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:09.028926 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 18:38:09.028954 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 18:38:09.029606 26191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a: Bootstrap starting.
I20260927 18:38:09.030146 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:09.030272 26191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:09.030568 26191 log.cc:824] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:09.031328 26191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a: No bootstrap required, opened a new log
I20260927 18:38:09.032640 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:09.031757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39391 (local address 127.0.0.1:43755)
0927 18:38:09.031794 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:09.031796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:09.031876 (+ 80us) server_negotiation.cc:407] Connection header received
0927 18:38:09.031933 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:09.031934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:09.031937 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:09.031976 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:09.032452 (+ 476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:09.032453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:09.032453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:09.032456 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:09.032467 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:09.032470 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:09.032503 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:09.032560 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 18:38:09.032568 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:38:09.033545 26191 raft_consensus.cc:348] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851855212c54d2db4c8cb3a7419340a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 18:38:09.033671 26191 raft_consensus.cc:374] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:09.033692 26191 raft_consensus.cc:729] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e851855212c54d2db4c8cb3a7419340a, State: Initialized, Role: FOLLOWER
I20260927 18:38:09.033766 26191 consensus_queue.cc:260] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851855212c54d2db4c8cb3a7419340a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 18:38:09.033823 26191 raft_consensus.cc:388] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:09.033841 26191 raft_consensus.cc:482] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:09.033870 26191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:09.034317 26191 raft_consensus.cc:504] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851855212c54d2db4c8cb3a7419340a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 18:38:09.034403 26191 leader_election.cc:302] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e851855212c54d2db4c8cb3a7419340a; no voters:
I20260927 18:38:09.034511 26191 leader_election.cc:288] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:09.034588 26201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:09.034739 26201 raft_consensus.cc:686] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [term 1 LEADER]: Becoming Leader. State: Replica: e851855212c54d2db4c8cb3a7419340a, State: Running, Role: LEADER
I20260927 18:38:09.034762 26191 sys_catalog.cc:564] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:09.034878 26201 consensus_queue.cc:237] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851855212c54d2db4c8cb3a7419340a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } }
I20260927 18:38:09.035233 26202 sys_catalog.cc:455] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e851855212c54d2db4c8cb3a7419340a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851855212c54d2db4c8cb3a7419340a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } } }
I20260927 18:38:09.035367 26202 sys_catalog.cc:458] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:09.035557 26203 sys_catalog.cc:455] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e851855212c54d2db4c8cb3a7419340a. Latest consensus state: current_term: 1 leader_uuid: "e851855212c54d2db4c8cb3a7419340a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851855212c54d2db4c8cb3a7419340a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43755 } } }
I20260927 18:38:09.035652 26203 sys_catalog.cc:458] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:09.035679 26209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:09.036264 26209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:09.037667 26209 catalog_manager.cc:1347] Generated new cluster ID: c9ca03a27bcf4df2a9560871959dcef5
I20260927 18:38:09.037715 26209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:09.052392 26209 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:09.052668 26209 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:09.061432 26209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e851855212c54d2db4c8cb3a7419340a: Generated new TSK 0
I20260927 18:38:09.061621 26209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:09.074385 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26123
2026-09-27T18:38:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:09 dist-test-slave-19nl krb5kdc[26222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:09 dist-test-slave-19nl krb5kdc[26222](info): set up 2 sockets
Sep 27 18:38:09 dist-test-slave-19nl krb5kdc[26222](info): commencing operation
krb5kdc: starting...
W20260927 18:38:10.070291 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:10 dist-test-slave-19nl krb5kdc[26222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:10.173463 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:10.249953 26235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:10.250177 26235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:10.250213 26235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:10.251457 26235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:10.251524 26235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:10.251545 26235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:10.251565 26235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:10.251596 26235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:10.253192 26235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:10.253450 26235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:10.253687 26235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:10.258852 26240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:10.258879 26243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:10.258880 26241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:10.259215 26235 server_base.cc:1034] running on GCE node
I20260927 18:38:10.259454 26235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:10.259732 26235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:10.260896 26235 hybrid_clock.cc:648] HybridClock initialized: now 1790534290260880 us; error 28 us; skew 500 ppm
Sep 27 18:38:10 dist-test-slave-19nl krb5kdc[26222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:10.263239 26235 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:10.263630 26235 webserver.cc:460] Webserver started at http://127.0.0.1:46459/ using document root <none> and password file <none>
I20260927 18:38:10.263836 26235 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:10.263907 26235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:10.264050 26235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:10.265000 26235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "b326bb55bb9c460ba4989ffd379f41c1"
format_stamp: "Formatted at 2026-09-27 18:38:10 on dist-test-slave-19nl"
server_key: "902add63019e21282f3232b27fe7e4ac"
server_key_iv: "58b0c224b495a93316b3aade876caaef"
server_key_version: "encryptionkey@0"
I20260927 18:38:10.265340 26235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "b326bb55bb9c460ba4989ffd379f41c1"
format_stamp: "Formatted at 2026-09-27 18:38:10 on dist-test-slave-19nl"
server_key: "902add63019e21282f3232b27fe7e4ac"
server_key_iv: "58b0c224b495a93316b3aade876caaef"
server_key_version: "encryptionkey@0"
I20260927 18:38:10.266590 26235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:10.267292 26250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:10.267457 26235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:10.267551 26235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "b326bb55bb9c460ba4989ffd379f41c1"
format_stamp: "Formatted at 2026-09-27 18:38:10 on dist-test-slave-19nl"
server_key: "902add63019e21282f3232b27fe7e4ac"
server_key_iv: "58b0c224b495a93316b3aade876caaef"
server_key_version: "encryptionkey@0"
I20260927 18:38:10.267635 26235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:10.274219 26235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:10.276264 26235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:10.276412 26235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:10.279927 26235 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37653
I20260927 18:38:10.279985 26302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37653 every 8 connection(s)
I20260927 18:38:10.280284 26235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:10.281118 26303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:10.283030 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1: Bootstrap starting.
I20260927 18:38:10.283481 26303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:10.283694 26303 log.cc:824] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:10.284242 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1: No bootstrap required, opened a new log
I20260927 18:38:10.285548 26303 raft_consensus.cc:348] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b326bb55bb9c460ba4989ffd379f41c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260927 18:38:10.285668 26303 raft_consensus.cc:374] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:10.285689 26303 raft_consensus.cc:729] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b326bb55bb9c460ba4989ffd379f41c1, State: Initialized, Role: FOLLOWER
I20260927 18:38:10.285801 26303 consensus_queue.cc:260] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b326bb55bb9c460ba4989ffd379f41c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260927 18:38:10.285873 26303 raft_consensus.cc:388] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:10.285892 26303 raft_consensus.cc:482] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:10.285918 26303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:10.286327 26303 raft_consensus.cc:504] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b326bb55bb9c460ba4989ffd379f41c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260927 18:38:10.286398 26303 leader_election.cc:302] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b326bb55bb9c460ba4989ffd379f41c1; no voters:
I20260927 18:38:10.286504 26303 leader_election.cc:288] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:10.286574 26306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:10.286720 26306 raft_consensus.cc:686] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [term 1 LEADER]: Becoming Leader. State: Replica: b326bb55bb9c460ba4989ffd379f41c1, State: Running, Role: LEADER
I20260927 18:38:10.286737 26303 sys_catalog.cc:564] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:10.286801 26306 consensus_queue.cc:237] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b326bb55bb9c460ba4989ffd379f41c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260927 18:38:10.286962 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26235
I20260927 18:38:10.287074 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:10.287267 5727 external_mini_cluster.cc:1428] Setting key ba00f7492bb40b020518189855cdce86
I20260927 18:38:10.287267 26308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b326bb55bb9c460ba4989ffd379f41c1. Latest consensus state: current_term: 1 leader_uuid: "b326bb55bb9c460ba4989ffd379f41c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b326bb55bb9c460ba4989ffd379f41c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260927 18:38:10.287334 26308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:10.287467 26307 sys_catalog.cc:455] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b326bb55bb9c460ba4989ffd379f41c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b326bb55bb9c460ba4989ffd379f41c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260927 18:38:10.287531 26307 sys_catalog.cc:458] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:10.287658 26310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:10.288149 26310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:10.289721 26310 catalog_manager.cc:1347] Generated new cluster ID: b636ae57e6f8423a845c529327fb97b6
I20260927 18:38:10.289779 26310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:38:10 dist-test-slave-19nl krb5kdc[26222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:38:10.299695 26310 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:10.300139 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:10.287690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58527 (local address 127.0.0.1:37653)
0927 18:38:10.287831 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:10.287840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:10.289260 (+ 1420us) server_negotiation.cc:407] Connection header received
0927 18:38:10.289408 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:10.289415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:10.289468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:10.289602 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:10.290438 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:10.291906 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:10.292962 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:10.293562 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:10.293656 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:10.296457 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:10.296460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:10.296461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:10.296471 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:10.298277 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:10.298665 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:10.298668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:10.298668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:10.298782 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:10.298973 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:10.298973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:10.298974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:10.299264 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:10.299413 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:10.299804 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 18:38:10.299932 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260927 18:38:10.300160 26310 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:10.301057 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:10.303771 26310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b326bb55bb9c460ba4989ffd379f41c1: Generated new TSK 0
I20260927 18:38:10.303920 26310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:10.309027 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:10.302782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53389 (local address 127.0.0.1:37653)
0927 18:38:10.302814 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:10.302816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:10.303063 (+ 247us) server_negotiation.cc:407] Connection header received
0927 18:38:10.303128 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:10.303129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:10.303139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:10.303195 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:10.303832 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:10.304569 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:10.305278 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:10.305588 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:10.305654 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:10.306727 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:10.306728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:10.306728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:10.306734 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:10.308077 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:10.308357 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:10.308358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:10.308358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:10.308411 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:10.308599 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:10.308600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:10.308600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:10.308717 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:10.308805 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:10.308876 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:38:10.308948 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:38:10.310838 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26235
2026-09-27T18:38:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:10 dist-test-slave-19nl krb5kdc[26334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:10 dist-test-slave-19nl krb5kdc[26334](info): set up 2 sockets
Sep 27 18:38:10 dist-test-slave-19nl krb5kdc[26334](info): commencing operation
krb5kdc: starting...
W20260927 18:38:11.282183 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:11 dist-test-slave-19nl krb5kdc[26334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:11.390610 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:11.466701 26347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:11.466881 26347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:11.466903 26347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:11.467972 26347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:11.468006 26347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:11.468019 26347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:11.468031 26347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:11.468043 26347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:11.469344 26347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:11.469512 26347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:11.469668 26347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:11.474078 26353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:11.474079 26355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:11.474123 26352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:11.474213 26347 server_base.cc:1034] running on GCE node
I20260927 18:38:11.474411 26347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:11.474695 26347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:11.475828 26347 hybrid_clock.cc:648] HybridClock initialized: now 1790534291475819 us; error 24 us; skew 500 ppm
Sep 27 18:38:11 dist-test-slave-19nl krb5kdc[26334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:11.477962 26347 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:11.478425 26347 webserver.cc:460] Webserver started at http://127.0.0.1:39421/ using document root <none> and password file <none>
I20260927 18:38:11.478680 26347 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:11.478744 26347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:11.478881 26347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:11.479812 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "fb50f2706cf348b4ab05c343d03e17fe"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-19nl"
server_key: "dc6ef73cdc22d449c9c58b2349ac85dd"
server_key_iv: "26e1cfc92e23546efd397a16e5ec5c32"
server_key_version: "encryptionkey@0"
I20260927 18:38:11.480135 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "fb50f2706cf348b4ab05c343d03e17fe"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-19nl"
server_key: "dc6ef73cdc22d449c9c58b2349ac85dd"
server_key_iv: "26e1cfc92e23546efd397a16e5ec5c32"
server_key_version: "encryptionkey@0"
I20260927 18:38:11.481266 26347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:11.481868 26362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:11.482015 26347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:11.482101 26347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "fb50f2706cf348b4ab05c343d03e17fe"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-19nl"
server_key: "dc6ef73cdc22d449c9c58b2349ac85dd"
server_key_iv: "26e1cfc92e23546efd397a16e5ec5c32"
server_key_version: "encryptionkey@0"
I20260927 18:38:11.482179 26347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:11.494980 26347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:11.496505 26347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:11.496688 26347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:11.500394 26347 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42655
I20260927 18:38:11.500422 26414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42655 every 8 connection(s)
I20260927 18:38:11.500741 26347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:11.501576 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:11.503531 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe: Bootstrap starting.
I20260927 18:38:11.504009 26415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:11.504249 26415 log.cc:824] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:11.504401 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26347
I20260927 18:38:11.504544 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:11.504755 5727 external_mini_cluster.cc:1428] Setting key f644dd16f608fe63e3efa1096386aff7
I20260927 18:38:11.504989 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe: No bootstrap required, opened a new log
I20260927 18:38:11.507728 26415 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb50f2706cf348b4ab05c343d03e17fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260927 18:38:11.507879 26415 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:11.507920 26415 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb50f2706cf348b4ab05c343d03e17fe, State: Initialized, Role: FOLLOWER
I20260927 18:38:11.508028 26415 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb50f2706cf348b4ab05c343d03e17fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260927 18:38:11.508096 26415 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:11.508143 26415 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:11.508196 26415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:11.508715 26415 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb50f2706cf348b4ab05c343d03e17fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260927 18:38:11.508795 26415 leader_election.cc:302] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb50f2706cf348b4ab05c343d03e17fe; no voters:
I20260927 18:38:11.508972 26415 leader_election.cc:288] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:11.508999 26420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:11.509104 26420 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [term 1 LEADER]: Becoming Leader. State: Replica: fb50f2706cf348b4ab05c343d03e17fe, State: Running, Role: LEADER
I20260927 18:38:11.509169 26420 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb50f2706cf348b4ab05c343d03e17fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260927 18:38:11.509217 26415 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:11.509461 26420 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb50f2706cf348b4ab05c343d03e17fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb50f2706cf348b4ab05c343d03e17fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } } }
I20260927 18:38:11.509518 26420 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:11.509641 26420 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb50f2706cf348b4ab05c343d03e17fe. Latest consensus state: current_term: 1 leader_uuid: "fb50f2706cf348b4ab05c343d03e17fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb50f2706cf348b4ab05c343d03e17fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } } }
I20260927 18:38:11.509709 26420 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe [sys.catalog]: This master's current role is: LEADER
W20260927 18:38:11.510941 26435 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:11.511039 26435 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:11.511224 26424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:11.511411 26424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:11.511441 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:11.505249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44561 (local address 127.0.0.1:42655)
0927 18:38:11.505423 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:11.505431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:11.505465 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:38:11.506285 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:11.506294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:11.506353 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:11.506491 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:11.506716 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.508117 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.508949 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.509638 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.509765 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:11.510526 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:11.510530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:11.510530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:11.510738 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:11.510776 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:11.510791 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:11.510846 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:11.511091 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 18:38:11.511171 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260927 18:38:11.512344 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:11.513865 26424 catalog_manager.cc:1347] Generated new cluster ID: 97042699fa9b4b9bbeebc8fd6f90204a
I20260927 18:38:11.513913 26424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:11.516911 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:11.513709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56189 (local address 127.0.0.1:42655)
0927 18:38:11.513752 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:11.513754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:11.513956 (+ 202us) server_negotiation.cc:407] Connection header received
0927 18:38:11.514028 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:11.514029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:11.514040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:11.514097 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:11.514216 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.514751 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.515729 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.515968 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.516045 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:11.516676 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:11.516677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:11.516677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:11.516682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:11.516694 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:11.516698 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:11.516744 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:11.516800 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:38:11.516818 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:38:11.540553 26424 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:11.541030 26424 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:11.543366 26424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb50f2706cf348b4ab05c343d03e17fe: Generated new TSK 0
I20260927 18:38:11.543526 26424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:11.551679 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26347
2026-09-27T18:38:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:11 dist-test-slave-19nl krb5kdc[26446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:11 dist-test-slave-19nl krb5kdc[26446](info): set up 2 sockets
Sep 27 18:38:11 dist-test-slave-19nl krb5kdc[26446](info): commencing operation
krb5kdc: starting...
W20260927 18:38:12.526647 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:12 dist-test-slave-19nl krb5kdc[26446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:12.634717 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:12.714092 26459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:12.714330 26459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:12.714367 26459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:12.715564 26459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:12.715646 26459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:12.715670 26459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:12.715692 26459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:12.715719 26459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:12.717190 26459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:12.717455 26459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:12.717677 26459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:12.722196 26465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:12.722306 26459 server_base.cc:1034] running on GCE node
W20260927 18:38:12.722383 26464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:12.722229 26467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:12.722718 26459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:12.722947 26459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:12.724103 26459 hybrid_clock.cc:648] HybridClock initialized: now 1790534292724092 us; error 28 us; skew 500 ppm
Sep 27 18:38:12 dist-test-slave-19nl krb5kdc[26446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:12.726445 26459 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:12.726874 26459 webserver.cc:460] Webserver started at http://127.0.0.1:36839/ using document root <none> and password file <none>
I20260927 18:38:12.727094 26459 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:12.727138 26459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:12.727296 26459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:12.728298 26459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "9bfe1043b9664c2aa7658ff17deeb916"
format_stamp: "Formatted at 2026-09-27 18:38:12 on dist-test-slave-19nl"
server_key: "19861d30bd8797526650da598ea4d968"
server_key_iv: "58d58746dbb945255710f429cfaf54ac"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.728662 26459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "9bfe1043b9664c2aa7658ff17deeb916"
format_stamp: "Formatted at 2026-09-27 18:38:12 on dist-test-slave-19nl"
server_key: "19861d30bd8797526650da598ea4d968"
server_key_iv: "58d58746dbb945255710f429cfaf54ac"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.729913 26459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:12.730669 26474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.730854 26459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.730923 26459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "9bfe1043b9664c2aa7658ff17deeb916"
format_stamp: "Formatted at 2026-09-27 18:38:12 on dist-test-slave-19nl"
server_key: "19861d30bd8797526650da598ea4d968"
server_key_iv: "58d58746dbb945255710f429cfaf54ac"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.731006 26459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:12.734086 26459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:12.735868 26459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:12.736014 26459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:12.739802 26459 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41859
I20260927 18:38:12.739964 26526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41859 every 8 connection(s)
I20260927 18:38:12.740181 26459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:12.741099 26527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:12.743320 26527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916: Bootstrap starting.
I20260927 18:38:12.743886 26527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:12.744171 26527 log.cc:824] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:12.744848 26527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916: No bootstrap required, opened a new log
I20260927 18:38:12.746363 26527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfe1043b9664c2aa7658ff17deeb916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260927 18:38:12.746493 26527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:12.746529 26527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfe1043b9664c2aa7658ff17deeb916, State: Initialized, Role: FOLLOWER
I20260927 18:38:12.746709 26527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfe1043b9664c2aa7658ff17deeb916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260927 18:38:12.746774 26527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:12.746796 26527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:12.746866 26527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:12.747395 26527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfe1043b9664c2aa7658ff17deeb916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260927 18:38:12.747479 26527 leader_election.cc:302] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bfe1043b9664c2aa7658ff17deeb916; no voters:
I20260927 18:38:12.747669 26527 leader_election.cc:288] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:12.747713 26530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:12.747880 26530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [term 1 LEADER]: Becoming Leader. State: Replica: 9bfe1043b9664c2aa7658ff17deeb916, State: Running, Role: LEADER
I20260927 18:38:12.747920 26527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:12.748063 26530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfe1043b9664c2aa7658ff17deeb916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260927 18:38:12.748612 26531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bfe1043b9664c2aa7658ff17deeb916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfe1043b9664c2aa7658ff17deeb916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260927 18:38:12.748631 26532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bfe1043b9664c2aa7658ff17deeb916. Latest consensus state: current_term: 1 leader_uuid: "9bfe1043b9664c2aa7658ff17deeb916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfe1043b9664c2aa7658ff17deeb916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260927 18:38:12.748857 26532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:12.749136 26531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:12.749162 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26459
I20260927 18:38:12.749277 26539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:12.749290 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:12.749500 5727 external_mini_cluster.cc:1428] Setting key 33ac371a97adbd784c7af073a48ef342
I20260927 18:38:12.749789 26539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:12.751328 26539 catalog_manager.cc:1347] Generated new cluster ID: 0ef39b9f99964378beaa716bdef49b25
I20260927 18:38:12.751436 26539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:12.752256 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.749953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41081 (local address 127.0.0.1:41859)
0927 18:38:12.750773 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.750782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.750877 (+ 95us) server_negotiation.cc:407] Connection header received
0927 18:38:12.750999 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.751007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.751037 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.751184 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:12.751856 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:12.751860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:12.751860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:12.751864 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:12.751901 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:12.751913 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:12.751974 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.752055 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.752091 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":74,"threads_started":1}
I20260927 18:38:12.753130 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:12.755988 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.754796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35345 (local address 127.0.0.1:41859)
0927 18:38:12.754830 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.754832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.755044 (+ 212us) server_negotiation.cc:407] Connection header received
0927 18:38:12.755132 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.755132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.755137 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.755163 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:12.755784 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:12.755786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:12.755786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:12.755790 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:12.755801 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:12.755805 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:12.755831 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.755922 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.755933 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:12.761139 26539 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:12.761279 26539 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:12.764638 26539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bfe1043b9664c2aa7658ff17deeb916: Generated new TSK 0
I20260927 18:38:12.764794 26539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:12.773160 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26459
2026-09-27T18:38:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:12 dist-test-slave-19nl krb5kdc[26558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:12 dist-test-slave-19nl krb5kdc[26558](info): set up 2 sockets
Sep 27 18:38:12 dist-test-slave-19nl krb5kdc[26558](info): commencing operation
krb5kdc: starting...
W20260927 18:38:13.762434 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:13 dist-test-slave-19nl krb5kdc[26558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:13.867369 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:13.943639 26571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:13.943851 26571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:13.943886 26571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:13.944983 26571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:13.945050 26571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:13.945071 26571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:13.945091 26571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:13.945108 26571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:13.946509 26571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:13.946794 26571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:13.947000 26571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:13.951643 26579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:13.951675 26576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:13.951675 26577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:13.951747 26571 server_base.cc:1034] running on GCE node
I20260927 18:38:13.952135 26571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:13.952409 26571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:13.953565 26571 hybrid_clock.cc:648] HybridClock initialized: now 1790534293953563 us; error 37 us; skew 500 ppm
Sep 27 18:38:13 dist-test-slave-19nl krb5kdc[26558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:13.956106 26571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:13.956539 26571 webserver.cc:460] Webserver started at http://127.0.0.1:46005/ using document root <none> and password file <none>
I20260927 18:38:13.956764 26571 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:13.956830 26571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:13.956976 26571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:13.957952 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "91328816653f42a8bc9c7b7f4b5eebb8"
format_stamp: "Formatted at 2026-09-27 18:38:13 on dist-test-slave-19nl"
server_key: "70dbfbfb558fa634ec7c7ab438460919"
server_key_iv: "493c29b44f2d1702cd19dce7b6ab14fa"
server_key_version: "encryptionkey@0"
I20260927 18:38:13.958300 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "91328816653f42a8bc9c7b7f4b5eebb8"
format_stamp: "Formatted at 2026-09-27 18:38:13 on dist-test-slave-19nl"
server_key: "70dbfbfb558fa634ec7c7ab438460919"
server_key_iv: "493c29b44f2d1702cd19dce7b6ab14fa"
server_key_version: "encryptionkey@0"
I20260927 18:38:13.959607 26571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:13.960297 26586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:13.960459 26571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:13.960547 26571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "91328816653f42a8bc9c7b7f4b5eebb8"
format_stamp: "Formatted at 2026-09-27 18:38:13 on dist-test-slave-19nl"
server_key: "70dbfbfb558fa634ec7c7ab438460919"
server_key_iv: "493c29b44f2d1702cd19dce7b6ab14fa"
server_key_version: "encryptionkey@0"
I20260927 18:38:13.960644 26571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:13.969534 26571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:13.971673 26571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:13.971828 26571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:13.975739 26571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33217
I20260927 18:38:13.975766 26638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33217 every 8 connection(s)
I20260927 18:38:13.976080 26571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:13.976955 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:13.979075 26639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8: Bootstrap starting.
I20260927 18:38:13.979563 26639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:13.979799 26639 log.cc:824] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:13.980538 26639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8: No bootstrap required, opened a new log
I20260927 18:38:13.981204 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26571
I20260927 18:38:13.981343 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:13.981573 5727 external_mini_cluster.cc:1428] Setting key 5af1d1d17fa58c1ec656509e126c2333
I20260927 18:38:13.983297 26639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33217 } }
I20260927 18:38:13.983486 26639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:13.983530 26639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91328816653f42a8bc9c7b7f4b5eebb8, State: Initialized, Role: FOLLOWER
I20260927 18:38:13.983693 26639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33217 } }
I20260927 18:38:13.983775 26639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:13.983860 26639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:13.983922 26639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:13.984613 26639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33217 } }
I20260927 18:38:13.984727 26639 leader_election.cc:302] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91328816653f42a8bc9c7b7f4b5eebb8; no voters:
I20260927 18:38:13.984884 26639 leader_election.cc:288] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:13.984964 26642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:13.985229 26642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [term 1 LEADER]: Becoming Leader. State: Replica: 91328816653f42a8bc9c7b7f4b5eebb8, State: Running, Role: LEADER
I20260927 18:38:13.985212 26639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:13.986056 26642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33217 } }
I20260927 18:38:13.986471 26643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33217 } } }
I20260927 18:38:13.986577 26643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:13.986786 26645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91328816653f42a8bc9c7b7f4b5eebb8. Latest consensus state: current_term: 1 leader_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91328816653f42a8bc9c7b7f4b5eebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33217 } } }
I20260927 18:38:13.986886 26645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:13.987020 26650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:13.987471 26650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:13.990252 26661 mvcc.cc:205] Tried to move back new op lower bound from 7334028468176261120 to 7334028468167385088. Current Snapshot: MvccSnapshot[applied={T|T < 7334028468176261120}]
I20260927 18:38:13.990731 26650 catalog_manager.cc:1347] Generated new cluster ID: 852a09e3affd428eb4feb41b2b508319
I20260927 18:38:13.990784 26650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:38:13 dist-test-slave-19nl krb5kdc[26558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:38:13.996647 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.985276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:44491 (local address 127.0.0.1:33217)
0927 18:38:13.985457 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.985466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.985495 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:38:13.985610 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.985617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.985674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.986552 (+ 878us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.986688 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.988065 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.989087 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.989641 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.989745 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.992681 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:13.992685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:13.992685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:13.992696 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:13.995012 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.995451 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.995454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.995454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.995566 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.995809 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.995810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.995810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.996076 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:13.996219 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.996358 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.996447 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260927 18:38:13.997570 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:14.005879 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.998891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:43239 (local address 127.0.0.1:33217)
0927 18:38:13.998949 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.998951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.999147 (+ 196us) server_negotiation.cc:407] Connection header received
0927 18:38:13.999189 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.999190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.999200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.999254 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.999796 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:14.000548 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:14.001459 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:14.001789 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:14.001856 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:14.003184 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:14.003185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:14.003186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:14.003192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:14.004716 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:14.005093 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:14.005094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:14.005095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:14.005153 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:14.005364 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:14.005365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:14.005365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:14.005485 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:14.005594 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:14.005710 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 18:38:14.005786 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:38:14.016463 26650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:14.016929 26650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:14.019055 26650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91328816653f42a8bc9c7b7f4b5eebb8: Generated new TSK 0
I20260927 18:38:14.019223 26650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:14.023509 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26571
W20260927 18:38:14.027297 26556 connection.cc:535] client connection to 127.0.0.1:33217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33217): Connection reset by peer (error 104)
2026-09-27T18:38:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:14 dist-test-slave-19nl krb5kdc[26670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:14 dist-test-slave-19nl krb5kdc[26670](info): set up 2 sockets
Sep 27 18:38:14 dist-test-slave-19nl krb5kdc[26670](info): commencing operation
krb5kdc: starting...
W20260927 18:38:14.994724 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:15 dist-test-slave-19nl krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:15.100955 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:15.175608 26683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:15.175787 26683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:15.175807 26683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:15.176903 26683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:15.176941 26683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:15.176955 26683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:15.177004 26683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:15.177030 26683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:15.178436 26683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:15.178691 26683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:15.178900 26683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:15.183213 26689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:15.183228 26691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:15.183228 26688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:15.183364 26683 server_base.cc:1034] running on GCE node
I20260927 18:38:15.183652 26683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:15.183921 26683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:15.185083 26683 hybrid_clock.cc:648] HybridClock initialized: now 1790534295185077 us; error 34 us; skew 500 ppm
Sep 27 18:38:15 dist-test-slave-19nl krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:15.187372 26683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:15.187801 26683 webserver.cc:460] Webserver started at http://127.0.0.1:44613/ using document root <none> and password file <none>
I20260927 18:38:15.188004 26683 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:15.188072 26683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:15.188202 26683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:15.189121 26683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed"
format_stamp: "Formatted at 2026-09-27 18:38:15 on dist-test-slave-19nl"
server_key: "b7b6143af3f3eedbe2e51b4c3ecebf01"
server_key_iv: "4537a74fab45027a5c70b9a112ae6cfb"
server_key_version: "encryptionkey@0"
I20260927 18:38:15.189450 26683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed"
format_stamp: "Formatted at 2026-09-27 18:38:15 on dist-test-slave-19nl"
server_key: "b7b6143af3f3eedbe2e51b4c3ecebf01"
server_key_iv: "4537a74fab45027a5c70b9a112ae6cfb"
server_key_version: "encryptionkey@0"
I20260927 18:38:15.190667 26683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:38:15.191279 26698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:15.191424 26683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:15.191509 26683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed"
format_stamp: "Formatted at 2026-09-27 18:38:15 on dist-test-slave-19nl"
server_key: "b7b6143af3f3eedbe2e51b4c3ecebf01"
server_key_iv: "4537a74fab45027a5c70b9a112ae6cfb"
server_key_version: "encryptionkey@0"
I20260927 18:38:15.191591 26683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:15.200855 26683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:15.202919 26683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:15.203081 26683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:15.206662 26683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41725
I20260927 18:38:15.206712 26750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41725 every 8 connection(s)
I20260927 18:38:15.207006 26683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:15.207808 26751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:15.209678 26751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed: Bootstrap starting.
I20260927 18:38:15.210196 26751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:15.210453 26751 log.cc:824] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:15.211079 26751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed: No bootstrap required, opened a new log
I20260927 18:38:15.212582 26751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260927 18:38:15.212699 26751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:15.212752 26751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce6b2e8f72e44d6ba9479dd700b5eed, State: Initialized, Role: FOLLOWER
I20260927 18:38:15.212901 26751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260927 18:38:15.212958 26751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:15.212978 26751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:15.213042 26751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:15.213517 26751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260927 18:38:15.213619 26751 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ce6b2e8f72e44d6ba9479dd700b5eed; no voters:
I20260927 18:38:15.213773 26751 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:15.213817 26754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:15.213971 26754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [term 1 LEADER]: Becoming Leader. State: Replica: 3ce6b2e8f72e44d6ba9479dd700b5eed, State: Running, Role: LEADER
I20260927 18:38:15.214114 26754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260927 18:38:15.214012 26751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:15.214501 26755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ce6b2e8f72e44d6ba9479dd700b5eed. Latest consensus state: current_term: 1 leader_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } } }
I20260927 18:38:15.214624 26756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce6b2e8f72e44d6ba9479dd700b5eed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } } }
I20260927 18:38:15.214623 26755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:15.214666 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26683
I20260927 18:38:15.214694 26756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:15.214776 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:15.214840 26758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:15.214957 5727 external_mini_cluster.cc:1428] Setting key 9d9c3e10d9d9c4f1c8cf316614e4952b
I20260927 18:38:15.215247 26758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:15.218026 26758 catalog_manager.cc:1347] Generated new cluster ID: 6793f0d60a9f471d9ba93b5c53a50096
I20260927 18:38:15.218093 26758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:15.221802 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:15.215386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:54245 (local address 127.0.0.1:41725)
0927 18:38:15.216677 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:15.216686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:15.216717 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:38:15.216825 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:15.216831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:15.216893 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:15.217023 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:15.217244 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:15.218692 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:15.219715 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:15.220308 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:15.220415 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:15.221022 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:15.221026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:15.221026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:15.221219 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:15.221255 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:15.221268 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:15.221348 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:15.221434 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 18:38:15.221459 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":62,"threads_started":1}
I20260927 18:38:15.222585 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:15.225540 26758 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:15.225986 26758 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:15.228451 26758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ce6b2e8f72e44d6ba9479dd700b5eed: Generated new TSK 0
I20260927 18:38:15.228608 26758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:15.228785 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:15.223963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:39379 (local address 127.0.0.1:41725)
0927 18:38:15.223992 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:15.223994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:15.225311 (+ 1317us) server_negotiation.cc:407] Connection header received
0927 18:38:15.225378 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:15.225379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:15.225389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:15.225435 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:15.225652 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:15.226365 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:15.227328 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:15.227736 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:15.227815 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:15.228485 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:15.228486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:15.228486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:15.228490 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:15.228502 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:15.228521 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:15.228571 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:15.228704 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 18:38:15.228717 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:38:15.230335 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26683
W20260927 18:38:15.234624 26668 connection.cc:535] client connection to 127.0.0.1:41725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41725): Connection reset by peer (error 104)
2026-09-27T18:38:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:15 dist-test-slave-19nl krb5kdc[26782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:15 dist-test-slave-19nl krb5kdc[26782](info): set up 2 sockets
Sep 27 18:38:15 dist-test-slave-19nl krb5kdc[26782](info): commencing operation
krb5kdc: starting...
W20260927 18:38:16.210434 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:16 dist-test-slave-19nl krb5kdc[26782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:16.315327 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:16.390702 26795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:16.390902 26795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:16.390934 26795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:16.391989 26795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:16.392046 26795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:16.392068 26795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:16.392100 26795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:16.392124 26795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:16.393422 26795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:16.393692 26795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:16.393899 26795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:16.398254 26803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:16.398288 26800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:16.398406 26795 server_base.cc:1034] running on GCE node
W20260927 18:38:16.398449 26801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:16.398710 26795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:16.398973 26795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:16.400110 26795 hybrid_clock.cc:648] HybridClock initialized: now 1790534296400091 us; error 35 us; skew 500 ppm
Sep 27 18:38:16 dist-test-slave-19nl krb5kdc[26782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:16.402335 26795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:16.402778 26795 webserver.cc:460] Webserver started at http://127.0.0.1:41931/ using document root <none> and password file <none>
I20260927 18:38:16.402985 26795 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:16.403024 26795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:16.403169 26795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:16.404088 26795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "08ea4f07d75a43e0a2e3f7c23221e50f"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-19nl"
server_key: "e2baf39471c587aa6fbcc46a2c0e9183"
server_key_iv: "22d8b5851f531e292bbc12b49b6ce2a9"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.404397 26795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "08ea4f07d75a43e0a2e3f7c23221e50f"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-19nl"
server_key: "e2baf39471c587aa6fbcc46a2c0e9183"
server_key_iv: "22d8b5851f531e292bbc12b49b6ce2a9"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.405507 26795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:16.406100 26810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:16.406253 26795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:16.406327 26795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "08ea4f07d75a43e0a2e3f7c23221e50f"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-19nl"
server_key: "e2baf39471c587aa6fbcc46a2c0e9183"
server_key_iv: "22d8b5851f531e292bbc12b49b6ce2a9"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.406373 26795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:16.409328 26795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:16.411072 26795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:16.411181 26795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:16.414675 26795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41843
I20260927 18:38:16.414719 26862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41843 every 8 connection(s)
I20260927 18:38:16.414994 26795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:16.415880 26863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:16.417873 26863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f: Bootstrap starting.
I20260927 18:38:16.418376 26863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:16.418679 26863 log.cc:824] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:16.419293 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26795
I20260927 18:38:16.419333 26863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f: No bootstrap required, opened a new log
I20260927 18:38:16.419416 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:16.419606 5727 external_mini_cluster.cc:1428] Setting key c890d9be5befad804596ee400624bba9
I20260927 18:38:16.421097 26863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260927 18:38:16.421243 26863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:16.421281 26863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ea4f07d75a43e0a2e3f7c23221e50f, State: Initialized, Role: FOLLOWER
I20260927 18:38:16.421413 26863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260927 18:38:16.421535 26863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:16.421568 26863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:16.421626 26863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:16.422371 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.420031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:48815 (local address 127.0.0.1:41843)
0927 18:38:16.420197 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:16.420205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:16.420234 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:38:16.421077 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:16.421083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:16.421105 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:16.421204 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:16.421916 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:16.421919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:16.421920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:16.421923 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:16.421968 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:16.421978 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:16.422040 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:16.422123 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 18:38:16.422153 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260927 18:38:16.422704 26863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260927 18:38:16.422793 26863 leader_election.cc:302] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ea4f07d75a43e0a2e3f7c23221e50f; no voters:
I20260927 18:38:16.422959 26863 leader_election.cc:288] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:16.422991 26868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:16.423141 26868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [term 1 LEADER]: Becoming Leader. State: Replica: 08ea4f07d75a43e0a2e3f7c23221e50f, State: Running, Role: LEADER
I20260927 18:38:16.423195 26863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:16.423228 26868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260927 18:38:16.423324 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:16.423539 26868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ea4f07d75a43e0a2e3f7c23221e50f. Latest consensus state: current_term: 1 leader_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } } }
I20260927 18:38:16.423614 26868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:16.423547 26870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ea4f07d75a43e0a2e3f7c23221e50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } } }
I20260927 18:38:16.423652 26870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:16.424381 26874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:16.424862 26874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:16.426100 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.424808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:59121 (local address 127.0.0.1:41843)
0927 18:38:16.424842 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:16.424844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:16.424967 (+ 123us) server_negotiation.cc:407] Connection header received
0927 18:38:16.425060 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:16.425060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:16.425067 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:16.425098 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:16.425893 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:16.425894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:16.425894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:16.425899 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:16.425911 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:16.425915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:16.425959 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:16.426018 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 18:38:16.426033 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 18:38:16.426304 26874 catalog_manager.cc:1347] Generated new cluster ID: 9625315c311446a19f0e8718127027f4
I20260927 18:38:16.426352 26874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:16.439452 26874 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:16.439621 26874 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:16.448640 26874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ea4f07d75a43e0a2e3f7c23221e50f: Generated new TSK 0
I20260927 18:38:16.448818 26874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:16.455850 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26795
2026-09-27T18:38:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:16 dist-test-slave-19nl krb5kdc[26894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:16 dist-test-slave-19nl krb5kdc[26894](info): set up 2 sockets
Sep 27 18:38:16 dist-test-slave-19nl krb5kdc[26894](info): commencing operation
krb5kdc: starting...
W20260927 18:38:17.434537 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:17 dist-test-slave-19nl krb5kdc[26894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:17.540997 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:17.615592 26907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:17.615789 26907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:17.615809 26907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:17.616849 26907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:17.616899 26907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:17.616912 26907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:17.616925 26907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:17.616935 26907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:17.618258 26907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:17.618424 26907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:17.618625 26907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:17.622941 26912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:17.622956 26915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:17.622949 26913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:17.623167 26907 server_base.cc:1034] running on GCE node
I20260927 18:38:17.623478 26907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:17.623768 26907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:17.624925 26907 hybrid_clock.cc:648] HybridClock initialized: now 1790534297624921 us; error 34 us; skew 500 ppm
Sep 27 18:38:17 dist-test-slave-19nl krb5kdc[26894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:17.627345 26907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:17.627702 26907 webserver.cc:460] Webserver started at http://127.0.0.1:38809/ using document root <none> and password file <none>
I20260927 18:38:17.627868 26907 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:17.627904 26907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:17.627985 26907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:17.628901 26907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "b938f95c96d6477bb6b9969d9e6d053f"
format_stamp: "Formatted at 2026-09-27 18:38:17 on dist-test-slave-19nl"
server_key: "e6f661c33a5fd1fd6a8c3ad58523c35c"
server_key_iv: "76c6cbe89ccf4471bc3333588c269f7f"
server_key_version: "encryptionkey@0"
I20260927 18:38:17.629185 26907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "b938f95c96d6477bb6b9969d9e6d053f"
format_stamp: "Formatted at 2026-09-27 18:38:17 on dist-test-slave-19nl"
server_key: "e6f661c33a5fd1fd6a8c3ad58523c35c"
server_key_iv: "76c6cbe89ccf4471bc3333588c269f7f"
server_key_version: "encryptionkey@0"
I20260927 18:38:17.630353 26907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:38:17.631111 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:17.631289 26907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 18:38:17.631377 26907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "b938f95c96d6477bb6b9969d9e6d053f"
format_stamp: "Formatted at 2026-09-27 18:38:17 on dist-test-slave-19nl"
server_key: "e6f661c33a5fd1fd6a8c3ad58523c35c"
server_key_iv: "76c6cbe89ccf4471bc3333588c269f7f"
server_key_version: "encryptionkey@0"
I20260927 18:38:17.631430 26907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:17.651939 26907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:17.653733 26907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:17.653841 26907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:17.657737 26907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40309
I20260927 18:38:17.657811 26974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40309 every 8 connection(s)
I20260927 18:38:17.658156 26907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:17.659137 26975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:17.661357 26975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f: Bootstrap starting.
I20260927 18:38:17.661948 26975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:17.662236 26975 log.cc:824] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:17.662967 26975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f: No bootstrap required, opened a new log
I20260927 18:38:17.664474 26975 raft_consensus.cc:348] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b938f95c96d6477bb6b9969d9e6d053f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260927 18:38:17.664624 26975 raft_consensus.cc:374] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:17.664670 26975 raft_consensus.cc:729] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b938f95c96d6477bb6b9969d9e6d053f, State: Initialized, Role: FOLLOWER
I20260927 18:38:17.664834 26975 consensus_queue.cc:260] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b938f95c96d6477bb6b9969d9e6d053f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260927 18:38:17.664968 26975 raft_consensus.cc:388] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:17.665024 26975 raft_consensus.cc:482] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:17.665014 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 26907
I20260927 18:38:17.665081 26975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:17.665156 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:17.665349 5727 external_mini_cluster.cc:1428] Setting key ccdc4be91075fbd740a610ffaf09e976
I20260927 18:38:17.665654 26975 raft_consensus.cc:504] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b938f95c96d6477bb6b9969d9e6d053f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260927 18:38:17.665802 26975 leader_election.cc:302] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b938f95c96d6477bb6b9969d9e6d053f; no voters:
I20260927 18:38:17.665971 26975 leader_election.cc:288] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:17.666028 26979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:17.666214 26979 raft_consensus.cc:686] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [term 1 LEADER]: Becoming Leader. State: Replica: b938f95c96d6477bb6b9969d9e6d053f, State: Running, Role: LEADER
I20260927 18:38:17.666263 26975 sys_catalog.cc:564] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:17.666334 26979 consensus_queue.cc:237] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b938f95c96d6477bb6b9969d9e6d053f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260927 18:38:17.666908 26981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b938f95c96d6477bb6b9969d9e6d053f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b938f95c96d6477bb6b9969d9e6d053f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } } }
I20260927 18:38:17.666990 26981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:17.667279 26989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:17.667339 26982 sys_catalog.cc:455] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b938f95c96d6477bb6b9969d9e6d053f. Latest consensus state: current_term: 1 leader_uuid: "b938f95c96d6477bb6b9969d9e6d053f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b938f95c96d6477bb6b9969d9e6d053f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } } }
I20260927 18:38:17.667469 26982 sys_catalog.cc:458] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:17.667790 26989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:17.669423 26989 catalog_manager.cc:1347] Generated new cluster ID: a49a814d6615408f828b7ca7b6c66929
I20260927 18:38:17.669472 26989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:38:17 dist-test-slave-19nl krb5kdc[26894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:38:17.676856 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.665850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:58931 (local address 127.0.0.1:40309)
0927 18:38:17.666051 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.666061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.666098 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:38:17.666213 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.666220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.666273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.667332 (+ 1059us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:17.667459 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.669000 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.669933 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.670554 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.670704 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.673349 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:17.673353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:17.673353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:17.673364 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:17.675232 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.675678 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.675681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.675681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.675787 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.676038 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.676038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.676039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.676300 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:17.676440 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.676577 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.676664 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260927 18:38:17.677879 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:17.685268 26989 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:17.685850 26989 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:17.686046 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.679256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:47745 (local address 127.0.0.1:40309)
0927 18:38:17.679287 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.679289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.679297 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:38:17.679339 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.679340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.679349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.679408 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:17.679944 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.680652 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.681554 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.681877 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.681944 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.683149 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:17.683151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:17.683151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:17.683157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:17.684626 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.685156 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.685157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.685157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.685215 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.685476 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.685477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.685478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.685613 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:17.685728 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.685850 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.685956 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
I20260927 18:38:17.689306 26989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b938f95c96d6477bb6b9969d9e6d053f: Generated new TSK 0
I20260927 18:38:17.689451 26989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:17.694835 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 26907
W20260927 18:38:17.698812 26892 connection.cc:535] client connection to 127.0.0.1:40309 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40309): Connection reset by peer (error 104)
2026-09-27T18:38:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:17 dist-test-slave-19nl krb5kdc[27006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:17 dist-test-slave-19nl krb5kdc[27006](info): set up 2 sockets
Sep 27 18:38:17 dist-test-slave-19nl krb5kdc[27006](info): commencing operation
krb5kdc: starting...
W20260927 18:38:18.662716 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:18 dist-test-slave-19nl krb5kdc[27006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:18.766921 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:18.841149 27019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:18.841360 27019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:18.841396 27019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:18.842461 27019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:18.842527 27019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:18.842553 27019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:18.842588 27019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:18.842617 27019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:18.844540 27019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:18.844779 27019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:18.844972 27019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:18.849335 27025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:18.849316 27027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:18.849310 27024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:18.849565 27019 server_base.cc:1034] running on GCE node
I20260927 18:38:18.849802 27019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:18.850113 27019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:18.851274 27019 hybrid_clock.cc:648] HybridClock initialized: now 1790534298851275 us; error 43 us; skew 500 ppm
Sep 27 18:38:18 dist-test-slave-19nl krb5kdc[27006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:18.856897 27019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:18.857234 27019 webserver.cc:460] Webserver started at http://127.0.0.1:37153/ using document root <none> and password file <none>
I20260927 18:38:18.857390 27019 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:18.857426 27019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:18.857539 27019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:18.858438 27019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "4afe8d23970e4d129dac8563fa193d32"
format_stamp: "Formatted at 2026-09-27 18:38:18 on dist-test-slave-19nl"
server_key: "8a6930720da6bac01f0d3efe5bfb23ea"
server_key_iv: "78f8bf39b6053f4f29df035f2ccc183c"
server_key_version: "encryptionkey@0"
I20260927 18:38:18.858783 27019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "4afe8d23970e4d129dac8563fa193d32"
format_stamp: "Formatted at 2026-09-27 18:38:18 on dist-test-slave-19nl"
server_key: "8a6930720da6bac01f0d3efe5bfb23ea"
server_key_iv: "78f8bf39b6053f4f29df035f2ccc183c"
server_key_version: "encryptionkey@0"
I20260927 18:38:18.859929 27019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:38:18.860577 27034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:18.860723 27019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:18.860826 27019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "4afe8d23970e4d129dac8563fa193d32"
format_stamp: "Formatted at 2026-09-27 18:38:18 on dist-test-slave-19nl"
server_key: "8a6930720da6bac01f0d3efe5bfb23ea"
server_key_iv: "78f8bf39b6053f4f29df035f2ccc183c"
server_key_version: "encryptionkey@0"
I20260927 18:38:18.860898 27019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:18.874008 27019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:18.875725 27019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:18.875869 27019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:18.879513 27019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46519
I20260927 18:38:18.879580 27086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46519 every 8 connection(s)
I20260927 18:38:18.879877 27019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:18.880663 27087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:18.881136 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27019
I20260927 18:38:18.881263 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:18.881441 5727 external_mini_cluster.cc:1428] Setting key a0431a58278c90ea352714d471d109c0
I20260927 18:38:18.883199 27087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32: Bootstrap starting.
I20260927 18:38:18.883728 27087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:18.883986 27087 log.cc:824] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:18.884642 27087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32: No bootstrap required, opened a new log
I20260927 18:38:18.886706 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.881929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:33213 (local address 127.0.0.1:46519)
0927 18:38:18.882101 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.882109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.882138 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:38:18.882505 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.882510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.882603 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.882751 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:18.882947 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.883870 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.884805 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.885368 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.885484 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.886026 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.886030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.886030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:18.886214 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.886249 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:18.886262 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.886349 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.886427 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.886449 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260927 18:38:18.887111 27087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afe8d23970e4d129dac8563fa193d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46519 } }
I20260927 18:38:18.887280 27087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:18.887329 27087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4afe8d23970e4d129dac8563fa193d32, State: Initialized, Role: FOLLOWER
I20260927 18:38:18.887439 27087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afe8d23970e4d129dac8563fa193d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46519 } }
I20260927 18:38:18.887516 27087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:18.887545 27087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:18.887593 27087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:18.887723 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:18.888155 27087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afe8d23970e4d129dac8563fa193d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46519 } }
I20260927 18:38:18.888259 27087 leader_election.cc:302] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4afe8d23970e4d129dac8563fa193d32; no voters:
I20260927 18:38:18.888404 27087 leader_election.cc:288] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:18.888432 27092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:18.888540 27092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [term 1 LEADER]: Becoming Leader. State: Replica: 4afe8d23970e4d129dac8563fa193d32, State: Running, Role: LEADER
I20260927 18:38:18.888638 27087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:18.888629 27092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afe8d23970e4d129dac8563fa193d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46519 } }
I20260927 18:38:18.889038 27092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4afe8d23970e4d129dac8563fa193d32. Latest consensus state: current_term: 1 leader_uuid: "4afe8d23970e4d129dac8563fa193d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afe8d23970e4d129dac8563fa193d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46519 } } }
I20260927 18:38:18.889108 27092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:18.889268 27093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4afe8d23970e4d129dac8563fa193d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afe8d23970e4d129dac8563fa193d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46519 } } }
I20260927 18:38:18.889336 27093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32 [sys.catalog]: This master's current role is: LEADER
W20260927 18:38:18.890466 27113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:18.890528 27113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:18.890617 27101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:18.890815 27101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:18.893165 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.889063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:37245 (local address 127.0.0.1:46519)
0927 18:38:18.889095 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.889097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.890343 (+ 1246us) server_negotiation.cc:407] Connection header received
0927 18:38:18.890389 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.890390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.890400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.890456 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:18.890677 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.891232 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.892041 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.892247 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.892319 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.892903 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.892904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.892904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:18.892908 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.892920 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:18.892924 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.892985 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.893044 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.893053 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:18.900422 27101 catalog_manager.cc:1347] Generated new cluster ID: 053b7cc72cf4404aa54cb05b6b0f700f
I20260927 18:38:18.900475 27101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:18.921398 27101 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:18.921921 27101 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:18.925760 27101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4afe8d23970e4d129dac8563fa193d32: Generated new TSK 0
I20260927 18:38:18.925931 27101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:18.927839 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27019
W20260927 18:38:18.931878 27004 connection.cc:535] client connection to 127.0.0.1:46519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46519): Connection reset by peer (error 104)
2026-09-27T18:38:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:18 dist-test-slave-19nl krb5kdc[27118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:18 dist-test-slave-19nl krb5kdc[27118](info): set up 2 sockets
Sep 27 18:38:18 dist-test-slave-19nl krb5kdc[27118](info): commencing operation
krb5kdc: starting...
W20260927 18:38:19.902702 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:19 dist-test-slave-19nl krb5kdc[27118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:38:20.029147 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:20.106295 27131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:20.106472 27131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:20.106493 27131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:20.107581 27131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:20.107621 27131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:20.107635 27131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:20.107669 27131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:20.107697 27131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:20.109016 27131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:20.109236 27131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:20.109452 27131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:20.113848 27137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:20.113859 27139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:20.113897 27136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:20.114001 27131 server_base.cc:1034] running on GCE node
I20260927 18:38:20.114274 27131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:20.114606 27131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:20.115765 27131 hybrid_clock.cc:648] HybridClock initialized: now 1790534300115748 us; error 28 us; skew 500 ppm
Sep 27 18:38:20 dist-test-slave-19nl krb5kdc[27118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:20.118090 27131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:38:20.118525 27131 webserver.cc:460] Webserver started at http://127.0.0.1:43773/ using document root <none> and password file <none>
I20260927 18:38:20.118791 27131 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:20.118860 27131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:20.119009 27131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:20.119975 27131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "604cd8e5ef5d41fe8859a1589a6efd80"
format_stamp: "Formatted at 2026-09-27 18:38:20 on dist-test-slave-19nl"
server_key: "8801c6c1fe1b4a84da154342ec1b0131"
server_key_iv: "f0ea7e203a83c162120eb08ef374b0e9"
server_key_version: "encryptionkey@0"
I20260927 18:38:20.120321 27131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "604cd8e5ef5d41fe8859a1589a6efd80"
format_stamp: "Formatted at 2026-09-27 18:38:20 on dist-test-slave-19nl"
server_key: "8801c6c1fe1b4a84da154342ec1b0131"
server_key_iv: "f0ea7e203a83c162120eb08ef374b0e9"
server_key_version: "encryptionkey@0"
I20260927 18:38:20.121587 27131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 18:38:20.122233 27146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:20.122400 27131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:20.122491 27131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "604cd8e5ef5d41fe8859a1589a6efd80"
format_stamp: "Formatted at 2026-09-27 18:38:20 on dist-test-slave-19nl"
server_key: "8801c6c1fe1b4a84da154342ec1b0131"
server_key_iv: "f0ea7e203a83c162120eb08ef374b0e9"
server_key_version: "encryptionkey@0"
I20260927 18:38:20.122604 27131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:20.125689 27131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:20.127473 27131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:20.127628 27131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:20.131383 27131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39473
I20260927 18:38:20.131429 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39473 every 8 connection(s)
I20260927 18:38:20.131742 27131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:20.132615 27199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:20.133119 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27131
I20260927 18:38:20.133253 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:20.133430 5727 external_mini_cluster.cc:1428] Setting key a22becebd43160aef03f6968c6312b1b
I20260927 18:38:20.135375 27199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80: Bootstrap starting.
I20260927 18:38:20.135639 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:20.133897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:42589 (local address 127.0.0.1:39473)
0927 18:38:20.134058 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:20.134065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:20.134087 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:38:20.134428 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:20.134433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:20.134454 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:20.134602 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:20.135202 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:20.135205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:20.135205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:20.135209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:20.135241 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:20.135251 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:20.135303 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:20.135385 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 18:38:20.135477 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260927 18:38:20.136061 27199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:20.136386 27199 log.cc:824] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:20.136555 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:20.137076 27199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80: No bootstrap required, opened a new log
I20260927 18:38:20.138953 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:20.138017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:39739 (local address 127.0.0.1:39473)
0927 18:38:20.138067 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:20.138068 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:20.138151 (+ 83us) server_negotiation.cc:407] Connection header received
0927 18:38:20.138204 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:20.138204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:20.138207 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:20.138250 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:20.138753 (+ 503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:20.138754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:20.138754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:20.138757 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:20.138767 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:20.138771 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:20.138811 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:20.138861 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 18:38:20.138870 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:38:20.139616 27199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260927 18:38:20.139755 27199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:20.139798 27199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604cd8e5ef5d41fe8859a1589a6efd80, State: Initialized, Role: FOLLOWER
I20260927 18:38:20.139905 27199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260927 18:38:20.140017 27199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:20.140057 27199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:20.140108 27199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:20.140602 27199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260927 18:38:20.140712 27199 leader_election.cc:302] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 604cd8e5ef5d41fe8859a1589a6efd80; no voters:
I20260927 18:38:20.140858 27199 leader_election.cc:288] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:20.140904 27209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:20.141083 27209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [term 1 LEADER]: Becoming Leader. State: Replica: 604cd8e5ef5d41fe8859a1589a6efd80, State: Running, Role: LEADER
I20260927 18:38:20.141114 27199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:20.141163 27209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260927 18:38:20.141511 27211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 604cd8e5ef5d41fe8859a1589a6efd80. Latest consensus state: current_term: 1 leader_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } } }
I20260927 18:38:20.141536 27210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604cd8e5ef5d41fe8859a1589a6efd80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } } }
I20260927 18:38:20.141610 27211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:20.141635 27210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:20.141806 27215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:20.142254 27215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:20.143666 27215 catalog_manager.cc:1347] Generated new cluster ID: 052dedc5853b40e7afcb33977ff1fbe5
I20260927 18:38:20.143718 27215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:20.154353 27215 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:20.154486 27215 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:20.159830 27215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 604cd8e5ef5d41fe8859a1589a6efd80: Generated new TSK 0
I20260927 18:38:20.159970 27215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:20.163192 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27131
2026-09-27T18:38:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:20 dist-test-slave-19nl krb5kdc[27230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:20 dist-test-slave-19nl krb5kdc[27230](info): set up 2 sockets
Sep 27 18:38:20 dist-test-slave-19nl krb5kdc[27230](info): commencing operation
krb5kdc: starting...
W20260927 18:38:21.142761 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:21 dist-test-slave-19nl krb5kdc[27230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:21.249522 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34009
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:21.327649 27243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:21.327847 27243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:21.327867 27243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:21.328878 27243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:21.328910 27243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:21.328924 27243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:21.328936 27243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:21.328948 27243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:21.330271 27243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:34009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:21.330451 27243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:21.330694 27243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:21.335256 27251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:21.335265 27248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:21.335265 27249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:21.335562 27243 server_base.cc:1034] running on GCE node
I20260927 18:38:21.335810 27243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:21.336133 27243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:21.337297 27243 hybrid_clock.cc:648] HybridClock initialized: now 1790534301337292 us; error 36 us; skew 500 ppm
Sep 27 18:38:21 dist-test-slave-19nl krb5kdc[27230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:21.339789 27243 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:21.340171 27243 webserver.cc:460] Webserver started at http://127.5.151.254:44273/ using document root <none> and password file <none>
I20260927 18:38:21.340353 27243 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:21.340391 27243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:21.340473 27243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:21.341418 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "5feee62ca7534d5499e5469ff28f9901"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-19nl"
server_key: "112783443595d14c10527191b40b149f"
server_key_iv: "1f7af171009048606371bae49c0eb81e"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.341702 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "5feee62ca7534d5499e5469ff28f9901"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-19nl"
server_key: "112783443595d14c10527191b40b149f"
server_key_iv: "1f7af171009048606371bae49c0eb81e"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.342976 27243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:21.343704 27258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:21.343899 27243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:21.343979 27243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "5feee62ca7534d5499e5469ff28f9901"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-19nl"
server_key: "112783443595d14c10527191b40b149f"
server_key_iv: "1f7af171009048606371bae49c0eb81e"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.344030 27243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:21.355723 27243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:21.357311 27243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:21.357410 27243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:21.360921 27243 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:34009
I20260927 18:38:21.360965 27310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:34009 every 8 connection(s)
I20260927 18:38:21.361284 27243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:21.362097 27311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:21.363483 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27243
I20260927 18:38:21.363612 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:21.363785 5727 external_mini_cluster.cc:1428] Setting key 3b0da96e1fbffb663a785bbb9e213eb5
I20260927 18:38:21.364665 27311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901: Bootstrap starting.
I20260927 18:38:21.365278 27311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:21.365613 27311 log.cc:824] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:21.366295 27311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901: No bootstrap required, opened a new log
I20260927 18:38:21.368525 27311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feee62ca7534d5499e5469ff28f9901" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34009 } }
I20260927 18:38:21.368691 27311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:21.368718 27311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5feee62ca7534d5499e5469ff28f9901, State: Initialized, Role: FOLLOWER
I20260927 18:38:21.368856 27311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feee62ca7534d5499e5469ff28f9901" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34009 } }
I20260927 18:38:21.368948 27311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:21.368980 27311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:21.369023 27311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:21.369731 27311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feee62ca7534d5499e5469ff28f9901" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34009 } }
I20260927 18:38:21.369831 27311 leader_election.cc:302] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5feee62ca7534d5499e5469ff28f9901; no voters:
I20260927 18:38:21.369990 27311 leader_election.cc:288] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:21.370023 27316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:21.370182 27316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [term 1 LEADER]: Becoming Leader. State: Replica: 5feee62ca7534d5499e5469ff28f9901, State: Running, Role: LEADER
I20260927 18:38:21.370232 27311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:21.370266 27316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feee62ca7534d5499e5469ff28f9901" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34009 } }
I20260927 18:38:21.370579 27317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5feee62ca7534d5499e5469ff28f9901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feee62ca7534d5499e5469ff28f9901" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34009 } } }
I20260927 18:38:21.370673 27317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [sys.catalog]: This master's current role is: LEADER
Sep 27 18:38:21 dist-test-slave-19nl krb5kdc[27230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:21.370832 27318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5feee62ca7534d5499e5469ff28f9901. Latest consensus state: current_term: 1 leader_uuid: "5feee62ca7534d5499e5469ff28f9901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feee62ca7534d5499e5469ff28f9901" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 34009 } } }
I20260927 18:38:21.370900 27318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901 [sys.catalog]: This master's current role is: LEADER
W20260927 18:38:21.372238 27331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:21.372298 27331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:21.372370 27323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:21.372514 27323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:21.373859 27323 catalog_manager.cc:1347] Generated new cluster ID: 06b3d3b45c514fefaa1a59bbe3a40e84
I20260927 18:38:21.373910 27323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:21.375038 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.364280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50631 (local address 127.5.151.254:34009)
0927 18:38:21.364427 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:21.364435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:21.364467 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:38:21.364855 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:21.364862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:21.364911 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:21.365014 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:21.365838 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.366880 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.367936 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.368321 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.368398 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.371271 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:21.371275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:21.371276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:21.371286 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:21.373324 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.373784 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.373786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.373786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.373881 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.374117 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.374118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.374118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.374389 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:21.374531 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:21.374772 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 18:38:21.374865 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260927 18:38:21.379317 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:21.387550 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.380827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53063 (local address 127.5.151.254:34009)
0927 18:38:21.380858 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:21.380860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:21.380915 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:38:21.380985 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:21.380986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:21.380994 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:21.381045 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:21.381632 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.382319 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.383305 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.383598 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.383654 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.384923 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:21.384924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:21.384925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:21.384931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:21.386333 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.386753 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.386754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.386754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.386803 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.387029 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.387030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.387030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.387160 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:21.387244 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:21.387411 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 18:38:21.387468 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 18:38:21.398726 27323 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:21.399411 27323 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:21.411613 27323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5feee62ca7534d5499e5469ff28f9901: Generated new TSK 0
I20260927 18:38:21.411787 27323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:21.420236 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27243
W20260927 18:38:21.424625 27228 connection.cc:535] client connection to 127.5.151.254:34009 recv error: Network error: failed to read from TLS socket (remote: 127.5.151.254:34009): Connection reset by peer (error 104)
2026-09-27T18:38:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:21 dist-test-slave-19nl krb5kdc[27342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:21 dist-test-slave-19nl krb5kdc[27342](info): set up 2 sockets
Sep 27 18:38:21 dist-test-slave-19nl krb5kdc[27342](info): commencing operation
krb5kdc: starting...
W20260927 18:38:22.414353 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:22 dist-test-slave-19nl krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:22.522967 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32997
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:22.597714 27355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:22.597891 27355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:22.597910 27355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:22.599014 27355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:22.599052 27355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:22.599066 27355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:22.599112 27355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:22.599141 27355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:22.600502 27355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:22.600723 27355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:22.600955 27355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:22.605484 27360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:22.605479 27361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:22.605554 27355 server_base.cc:1034] running on GCE node
W20260927 18:38:22.605486 27363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:22.605893 27355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:22.606173 27355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:22.607303 27355 hybrid_clock.cc:648] HybridClock initialized: now 1790534302607293 us; error 26 us; skew 500 ppm
Sep 27 18:38:22 dist-test-slave-19nl krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534302, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:22.609676 27355 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:22.610092 27355 webserver.cc:460] Webserver started at http://127.5.151.254:40561/ using document root <none> and password file <none>
I20260927 18:38:22.610266 27355 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:22.610311 27355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:22.610394 27355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:22.611357 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "fc7a440f0fd24424bee4a88cb33e71bd"
format_stamp: "Formatted at 2026-09-27 18:38:22 on dist-test-slave-19nl"
server_key: "a1f63013725c6cc05cf32283ee9b0462"
server_key_iv: "bdf7e3dee8542497f6bee93b49a0a8d0"
server_key_version: "encryptionkey@0"
I20260927 18:38:22.611635 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "fc7a440f0fd24424bee4a88cb33e71bd"
format_stamp: "Formatted at 2026-09-27 18:38:22 on dist-test-slave-19nl"
server_key: "a1f63013725c6cc05cf32283ee9b0462"
server_key_iv: "bdf7e3dee8542497f6bee93b49a0a8d0"
server_key_version: "encryptionkey@0"
I20260927 18:38:22.612792 27355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:38:22.613426 27370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:22.613595 27355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:22.613701 27355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "fc7a440f0fd24424bee4a88cb33e71bd"
format_stamp: "Formatted at 2026-09-27 18:38:22 on dist-test-slave-19nl"
server_key: "a1f63013725c6cc05cf32283ee9b0462"
server_key_iv: "bdf7e3dee8542497f6bee93b49a0a8d0"
server_key_version: "encryptionkey@0"
I20260927 18:38:22.613776 27355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:22.632938 27355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:22.635126 27355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:22.635279 27355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:22.639384 27355 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:32997
I20260927 18:38:22.639431 27422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:32997 every 8 connection(s)
I20260927 18:38:22.639765 27355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:22.640558 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:22.642699 27423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd: Bootstrap starting.
I20260927 18:38:22.643204 27423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:22.643424 27423 log.cc:824] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:22.644035 27423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd: No bootstrap required, opened a new log
I20260927 18:38:22.645465 27423 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32997 } }
I20260927 18:38:22.645586 27423 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:22.645605 27423 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7a440f0fd24424bee4a88cb33e71bd, State: Initialized, Role: FOLLOWER
I20260927 18:38:22.645696 27423 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32997 } }
I20260927 18:38:22.645768 27423 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:22.645798 27423 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:22.645849 27423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:22.646610 27423 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32997 } }
I20260927 18:38:22.646723 27423 leader_election.cc:302] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc7a440f0fd24424bee4a88cb33e71bd; no voters:
I20260927 18:38:22.646888 27423 leader_election.cc:288] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:22.646927 27426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:22.647063 27426 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [term 1 LEADER]: Becoming Leader. State: Replica: fc7a440f0fd24424bee4a88cb33e71bd, State: Running, Role: LEADER
I20260927 18:38:22.647130 27423 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:22.647186 27426 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32997 } }
I20260927 18:38:22.647430 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27355
I20260927 18:38:22.647533 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:22.647713 5727 external_mini_cluster.cc:1428] Setting key 8bdc1a39587646ea76d908a9c4b12e48
I20260927 18:38:22.648252 27427 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32997 } } }
I20260927 18:38:22.648463 27427 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:22.648360 27428 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc7a440f0fd24424bee4a88cb33e71bd. Latest consensus state: current_term: 1 leader_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7a440f0fd24424bee4a88cb33e71bd" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32997 } } }
I20260927 18:38:22.648835 27428 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd [sys.catalog]: This master's current role is: LEADER
W20260927 18:38:22.648831 27442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:22.649016 27442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:22.649184 27444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:22.649387 27444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:22.650924 27444 catalog_manager.cc:1347] Generated new cluster ID: 28c5b9f084594037b7c1bbc82185ef80
I20260927 18:38:22.650975 27444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:22.654214 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.648154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54501 (local address 127.5.151.254:32997)
0927 18:38:22.648319 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.648346 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.648742 (+ 396us) server_negotiation.cc:407] Connection header received
0927 18:38:22.649242 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.649255 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.649326 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.649429 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:22.649771 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.651102 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.652209 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.652761 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.652861 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.653572 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:22.653576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:22.653576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:22.653763 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:22.653798 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:22.653811 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:22.653892 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.653989 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.654013 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260927 18:38:22.655277 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:22.659917 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.656675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46639 (local address 127.5.151.254:32997)
0927 18:38:22.656712 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.656714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.656851 (+ 137us) server_negotiation.cc:407] Connection header received
0927 18:38:22.656907 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.656908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.656918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.656981 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:22.657092 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.657742 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.658710 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.658995 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.659051 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.659664 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:22.659665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:22.659665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:22.659669 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:22.659699 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:22.659703 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:22.659753 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.659848 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.659858 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:38:22.664209 27444 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:22.664913 27444 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:22.669550 27444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc7a440f0fd24424bee4a88cb33e71bd: Generated new TSK 0
I20260927 18:38:22.669730 27444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:22.671715 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27355
W20260927 18:38:22.675751 27340 connection.cc:535] client connection to 127.5.151.254:32997 recv error: Network error: failed to read from TLS socket (remote: 127.5.151.254:32997): Connection reset by peer (error 104)
2026-09-27T18:38:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:22 dist-test-slave-19nl krb5kdc[27454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:22 dist-test-slave-19nl krb5kdc[27454](info): set up 2 sockets
Sep 27 18:38:22 dist-test-slave-19nl krb5kdc[27454](info): commencing operation
krb5kdc: starting...
W20260927 18:38:23.646477 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:23 dist-test-slave-19nl krb5kdc[27454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:23.757341 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:37515
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:37515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:23.831696 27467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:23.831875 27467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:23.831897 27467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:23.832974 27467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:23.833012 27467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:23.833026 27467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:23.833040 27467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:23.833084 27467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:23.834376 27467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:37515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:37515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:23.834612 27467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:23.834810 27467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:23.839123 27473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:23.839115 27472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:23.839116 27475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:23.839352 27467 server_base.cc:1034] running on GCE node
I20260927 18:38:23.839587 27467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:23.839861 27467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:23.841010 27467 hybrid_clock.cc:648] HybridClock initialized: now 1790534303841006 us; error 49 us; skew 500 ppm
Sep 27 18:38:23 dist-test-slave-19nl krb5kdc[27454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534303, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:23.843250 27467 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:23.843670 27467 webserver.cc:460] Webserver started at http://127.5.151.254:45487/ using document root <none> and password file <none>
I20260927 18:38:23.843886 27467 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:23.843927 27467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:23.844063 27467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:23.844985 27467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "2339c951bc40466c91961adce2ed1730"
format_stamp: "Formatted at 2026-09-27 18:38:23 on dist-test-slave-19nl"
server_key: "900ceb86bc35a6f0cecbe36e4f016da1"
server_key_iv: "640de878bd9fa15c3aec04df788be34e"
server_key_version: "encryptionkey@0"
I20260927 18:38:23.845341 27467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "2339c951bc40466c91961adce2ed1730"
format_stamp: "Formatted at 2026-09-27 18:38:23 on dist-test-slave-19nl"
server_key: "900ceb86bc35a6f0cecbe36e4f016da1"
server_key_iv: "640de878bd9fa15c3aec04df788be34e"
server_key_version: "encryptionkey@0"
I20260927 18:38:23.846537 27467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 18:38:23.847275 27482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:23.847458 27467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:23.847524 27467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "2339c951bc40466c91961adce2ed1730"
format_stamp: "Formatted at 2026-09-27 18:38:23 on dist-test-slave-19nl"
server_key: "900ceb86bc35a6f0cecbe36e4f016da1"
server_key_iv: "640de878bd9fa15c3aec04df788be34e"
server_key_version: "encryptionkey@0"
I20260927 18:38:23.847613 27467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:23.850571 27467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:23.852159 27467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:23.852291 27467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:23.856006 27467 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:37515
I20260927 18:38:23.856056 27534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:37515 every 8 connection(s)
I20260927 18:38:23.856321 27467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:23.857228 27535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:23.859305 27535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730: Bootstrap starting.
I20260927 18:38:23.859815 27535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:23.860082 27535 log.cc:824] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:23.860692 27535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730: No bootstrap required, opened a new log
I20260927 18:38:23.860744 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27467
I20260927 18:38:23.860864 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:23.861042 5727 external_mini_cluster.cc:1428] Setting key ba26c1ac961f8cdae4e1c944652b478b
I20260927 18:38:23.862536 27535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2339c951bc40466c91961adce2ed1730" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37515 } }
I20260927 18:38:23.862749 27535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:23.862826 27535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2339c951bc40466c91961adce2ed1730, State: Initialized, Role: FOLLOWER
I20260927 18:38:23.862977 27535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2339c951bc40466c91961adce2ed1730" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37515 } }
I20260927 18:38:23.863082 27535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:23.863121 27535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:23.863169 27535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:23.863945 27535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2339c951bc40466c91961adce2ed1730" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37515 } }
I20260927 18:38:23.864051 27535 leader_election.cc:302] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2339c951bc40466c91961adce2ed1730; no voters:
I20260927 18:38:23.864221 27535 leader_election.cc:288] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:23.864272 27540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:23.864413 27540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [term 1 LEADER]: Becoming Leader. State: Replica: 2339c951bc40466c91961adce2ed1730, State: Running, Role: LEADER
I20260927 18:38:23.864454 27535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:23.864544 27540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2339c951bc40466c91961adce2ed1730" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37515 } }
I20260927 18:38:23.864900 27541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2339c951bc40466c91961adce2ed1730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2339c951bc40466c91961adce2ed1730" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37515 } } }
I20260927 18:38:23.864995 27541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:23.865128 27542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2339c951bc40466c91961adce2ed1730. Latest consensus state: current_term: 1 leader_uuid: "2339c951bc40466c91961adce2ed1730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2339c951bc40466c91961adce2ed1730" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 37515 } } }
I20260927 18:38:23.865358 27542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:23.865386 27549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:23.865875 27549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:23.867318 27549 catalog_manager.cc:1347] Generated new cluster ID: 5aee8f500ee84c2da8e19b1968ea6209
I20260927 18:38:23.867384 27549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:23.868798 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.861497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53091 (local address 127.5.151.254:37515)
0927 18:38:23.861666 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.861673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.861694 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:38:23.862520 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.862526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.862548 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.862715 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:23.868367 (+ 5652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:23.868371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:23.868371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:23.868375 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:23.868418 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:23.868433 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:23.868502 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.868578 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.868616 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260927 18:38:23.869740 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:23.872434 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.871249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52305 (local address 127.5.151.254:37515)
0927 18:38:23.871280 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.871282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.871486 (+ 204us) server_negotiation.cc:407] Connection header received
0927 18:38:23.871587 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.871588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.871593 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.871622 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:23.872211 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:23.872212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:23.872212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:23.872216 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:23.872227 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:23.872230 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:23.872256 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.872350 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.872362 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 18:38:23.876200 27549 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:23.876363 27549 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:23.880846 27549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2339c951bc40466c91961adce2ed1730: Generated new TSK 0
I20260927 18:38:23.881001 27549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:23.887782 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27467
2026-09-27T18:38:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:23 dist-test-slave-19nl krb5kdc[27566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:23 dist-test-slave-19nl krb5kdc[27566](info): set up 2 sockets
Sep 27 18:38:23 dist-test-slave-19nl krb5kdc[27566](info): commencing operation
krb5kdc: starting...
W20260927 18:38:24.886735 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:24 dist-test-slave-19nl krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:24.992959 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45797
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:42561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:45797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:25.067914 27579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:25.068126 27579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:25.068164 27579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:25.069237 27579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:25.069310 27579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:25.069334 27579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:25.069360 27579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:25.069381 27579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:25.070783 27579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:42561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:45797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:45797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:25.071043 27579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:25.071244 27579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:25.075546 27585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:25.075683 27587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:25.075558 27584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:25.075775 27579 server_base.cc:1034] running on GCE node
I20260927 18:38:25.075991 27579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:25.076253 27579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:25.077414 27579 hybrid_clock.cc:648] HybridClock initialized: now 1790534305077400 us; error 27 us; skew 500 ppm
Sep 27 18:38:25 dist-test-slave-19nl krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534305, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:25.079706 27579 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:25.080143 27579 webserver.cc:460] Webserver started at http://127.5.151.254:44693/ using document root <none> and password file <none>
I20260927 18:38:25.080354 27579 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:25.080417 27579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:25.080564 27579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:25.081508 27579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8"
format_stamp: "Formatted at 2026-09-27 18:38:25 on dist-test-slave-19nl"
server_key: "fabe8182d7d7761bd2d8db40071602c3"
server_key_iv: "7aef96ff53cea92f0fe78a02a616c968"
server_key_version: "encryptionkey@0"
I20260927 18:38:25.081836 27579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8"
format_stamp: "Formatted at 2026-09-27 18:38:25 on dist-test-slave-19nl"
server_key: "fabe8182d7d7761bd2d8db40071602c3"
server_key_iv: "7aef96ff53cea92f0fe78a02a616c968"
server_key_version: "encryptionkey@0"
I20260927 18:38:25.083163 27579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:25.083825 27594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:25.083985 27579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:25.084082 27579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8"
format_stamp: "Formatted at 2026-09-27 18:38:25 on dist-test-slave-19nl"
server_key: "fabe8182d7d7761bd2d8db40071602c3"
server_key_iv: "7aef96ff53cea92f0fe78a02a616c968"
server_key_version: "encryptionkey@0"
I20260927 18:38:25.084152 27579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:25.092844 27579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:25.094389 27579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:25.094538 27579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:25.098273 27579 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:45797
I20260927 18:38:25.098300 27646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:45797 every 8 connection(s)
I20260927 18:38:25.098685 27579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:25.099484 27647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:25.101521 27647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8: Bootstrap starting.
I20260927 18:38:25.102082 27647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:25.102366 27647 log.cc:824] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:25.103065 27647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8: No bootstrap required, opened a new log
I20260927 18:38:25.104461 27647 raft_consensus.cc:348] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45797 } }
I20260927 18:38:25.104607 27647 raft_consensus.cc:374] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:25.104652 27647 raft_consensus.cc:729] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5b42a1f86e4c8aa040d70d6f3b28b8, State: Initialized, Role: FOLLOWER
I20260927 18:38:25.104835 27647 consensus_queue.cc:260] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45797 } }
I20260927 18:38:25.104925 27647 raft_consensus.cc:388] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:25.104964 27647 raft_consensus.cc:482] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:25.105026 27647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:25.105767 27647 raft_consensus.cc:504] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45797 } }
I20260927 18:38:25.105892 27647 leader_election.cc:302] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be5b42a1f86e4c8aa040d70d6f3b28b8; no voters:
I20260927 18:38:25.106060 27647 leader_election.cc:288] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:25.106089 27650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:25.106290 27650 raft_consensus.cc:686] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [term 1 LEADER]: Becoming Leader. State: Replica: be5b42a1f86e4c8aa040d70d6f3b28b8, State: Running, Role: LEADER
I20260927 18:38:25.106318 27647 sys_catalog.cc:564] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:25.106413 27650 consensus_queue.cc:237] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45797 } }
I20260927 18:38:25.106737 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27579
I20260927 18:38:25.106786 27651 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45797 } } }
I20260927 18:38:25.106849 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:25.106854 27651 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:25.107040 5727 external_mini_cluster.cc:1428] Setting key d094aba8fdfd5c31f8f2f16a2d3c28e9
I20260927 18:38:25.107009 27652 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be5b42a1f86e4c8aa040d70d6f3b28b8. Latest consensus state: current_term: 1 leader_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5b42a1f86e4c8aa040d70d6f3b28b8" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 45797 } } }
I20260927 18:38:25.107071 27652 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:25.107177 27657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:25.107686 27657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:25.109243 27657 catalog_manager.cc:1347] Generated new cluster ID: 2430d561ac9446be948032561acbe1b6
I20260927 18:38:25.109293 27657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 18:38:25 dist-test-slave-19nl krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:25.120781 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:25.107499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57791 (local address 127.5.151.254:45797)
0927 18:38:25.107676 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:25.107798 (+ 122us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:25.110883 (+ 3085us) server_negotiation.cc:407] Connection header received
0927 18:38:25.110988 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:25.110994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:25.111056 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:25.111185 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:25.111814 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.113174 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.114171 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.114786 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.114890 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:25.117539 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:25.117543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:25.117543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:25.117560 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:25.119319 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:25.119688 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:25.119691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:25.119691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:25.119783 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:25.119975 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:25.119976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:25.119976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:25.120223 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:25.120334 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:25.120458 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 18:38:25.120576 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":57,"threads_started":1}
I20260927 18:38:25.121541 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:25.128248 27657 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:25.128849 27657 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:25.129626 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:25.122869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43905 (local address 127.5.151.254:45797)
0927 18:38:25.122905 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:25.122907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:25.123058 (+ 151us) server_negotiation.cc:407] Connection header received
0927 18:38:25.123114 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:25.123115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:25.123125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:25.123175 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:25.123743 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.124411 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.125363 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.125638 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.125693 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:25.126985 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:25.126986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:25.126986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:25.126992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:25.128481 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:25.128895 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:25.128896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:25.128896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:25.128954 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:25.129149 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:25.129150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:25.129150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:25.129253 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:25.129335 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:25.129461 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:38:25.129550 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:38:25.132937 27657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be5b42a1f86e4c8aa040d70d6f3b28b8: Generated new TSK 0
I20260927 18:38:25.133102 27657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:25.137065 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27579
2026-09-27T18:38:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:25 dist-test-slave-19nl krb5kdc[27678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:25 dist-test-slave-19nl krb5kdc[27678](info): set up 2 sockets
Sep 27 18:38:25 dist-test-slave-19nl krb5kdc[27678](info): commencing operation
krb5kdc: starting...
W20260927 18:38:26.130790 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:26 dist-test-slave-19nl krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:26.237162 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32937
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:26.314168 27691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:26.314339 27691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:26.314360 27691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:26.316181 27691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:26.316300 27691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:26.316334 27691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:26.316371 27691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:26.316397 27691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:26.317772 27691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:26.317997 27691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:26.318212 27691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:26.322611 27699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:26.322619 27697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:26.322602 27696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:26.322749 27691 server_base.cc:1034] running on GCE node
I20260927 18:38:26.323035 27691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:26.323251 27691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:26.324385 27691 hybrid_clock.cc:648] HybridClock initialized: now 1790534306324374 us; error 26 us; skew 500 ppm
Sep 27 18:38:26 dist-test-slave-19nl krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534306, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:26.326681 27691 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:26.327123 27691 webserver.cc:460] Webserver started at http://127.5.151.254:42107/ using document root <none> and password file <none>
I20260927 18:38:26.327327 27691 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:26.327390 27691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:26.327514 27691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:26.328418 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "86ff6cfe657f41d29fe6f87d83a8c210"
format_stamp: "Formatted at 2026-09-27 18:38:26 on dist-test-slave-19nl"
server_key: "90c4e006540880dc8f224db03364e650"
server_key_iv: "0b2e2dad47d866ea996b09d3c9cdec82"
server_key_version: "encryptionkey@0"
I20260927 18:38:26.328748 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "86ff6cfe657f41d29fe6f87d83a8c210"
format_stamp: "Formatted at 2026-09-27 18:38:26 on dist-test-slave-19nl"
server_key: "90c4e006540880dc8f224db03364e650"
server_key_iv: "0b2e2dad47d866ea996b09d3c9cdec82"
server_key_version: "encryptionkey@0"
I20260927 18:38:26.329950 27691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:26.330747 27706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:26.331023 27691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:38:26.331116 27691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "86ff6cfe657f41d29fe6f87d83a8c210"
format_stamp: "Formatted at 2026-09-27 18:38:26 on dist-test-slave-19nl"
server_key: "90c4e006540880dc8f224db03364e650"
server_key_iv: "0b2e2dad47d866ea996b09d3c9cdec82"
server_key_version: "encryptionkey@0"
I20260927 18:38:26.331197 27691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:26.363979 27691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:26.365885 27691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:26.366056 27691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:26.369801 27691 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:32937
I20260927 18:38:26.369828 27758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:32937 every 8 connection(s)
I20260927 18:38:26.370146 27691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:26.370975 27759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:26.371169 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27691
I20260927 18:38:26.371284 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:26.371443 5727 external_mini_cluster.cc:1428] Setting key baeeca2c7e22aaf6a508679a194ecc7a
I20260927 18:38:26.374243 27759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210: Bootstrap starting.
I20260927 18:38:26.374902 27759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:26.375178 27759 log.cc:824] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:26.375772 27759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210: No bootstrap required, opened a new log
I20260927 18:38:26.376993 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:26.371883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52325 (local address 127.5.151.254:32937)
0927 18:38:26.372070 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:26.372078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:26.372109 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:38:26.372509 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:26.372516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:26.372580 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:26.372745 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:26.372890 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:26.374205 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:26.375066 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:26.375504 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:26.375627 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:26.376307 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:26.376311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:26.376312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:26.376462 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:26.376488 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:26.376498 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:26.376572 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:26.376653 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 18:38:26.376714 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260927 18:38:26.377211 27759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32937 } }
I20260927 18:38:26.377355 27759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:26.377395 27759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ff6cfe657f41d29fe6f87d83a8c210, State: Initialized, Role: FOLLOWER
I20260927 18:38:26.377522 27759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32937 } }
I20260927 18:38:26.377599 27759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:26.377638 27759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:26.377679 27759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:26.377895 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:26.378255 27759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32937 } }
I20260927 18:38:26.378352 27759 leader_election.cc:302] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ff6cfe657f41d29fe6f87d83a8c210; no voters:
I20260927 18:38:26.378509 27759 leader_election.cc:288] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:26.378537 27764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:26.378705 27764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [term 1 LEADER]: Becoming Leader. State: Replica: 86ff6cfe657f41d29fe6f87d83a8c210, State: Running, Role: LEADER
I20260927 18:38:26.378757 27759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:26.378865 27764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32937 } }
I20260927 18:38:26.379240 27766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32937 } } }
I20260927 18:38:26.379324 27766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:26.379449 27766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86ff6cfe657f41d29fe6f87d83a8c210. Latest consensus state: current_term: 1 leader_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ff6cfe657f41d29fe6f87d83a8c210" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32937 } } }
I20260927 18:38:26.379498 27766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:26.379941 27778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:26.380441 27778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:26.381898 27778 catalog_manager.cc:1347] Generated new cluster ID: b58ee8c21a0240a4b0ef6dbee2d1b910
I20260927 18:38:26.381944 27778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:26.382929 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:26.379270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55627 (local address 127.5.151.254:32937)
0927 18:38:26.379421 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:26.379423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:26.379764 (+ 341us) server_negotiation.cc:407] Connection header received
0927 18:38:26.379838 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:26.379839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:26.379849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:26.379899 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:26.380068 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:26.380690 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:26.381648 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:26.381921 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:26.381990 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:26.382703 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:26.382704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:26.382705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:26.382708 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:26.382721 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:26.382724 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:26.382775 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:26.382850 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 18:38:26.382868 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 18:38:26.395664 27778 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:26.396322 27778 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:26.398600 27778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86ff6cfe657f41d29fe6f87d83a8c210: Generated new TSK 0
I20260927 18:38:26.398784 27778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:26.401726 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27691
2026-09-27T18:38:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:26 dist-test-slave-19nl krb5kdc[27790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:26 dist-test-slave-19nl krb5kdc[27790](info): set up 2 sockets
Sep 27 18:38:26 dist-test-slave-19nl krb5kdc[27790](info): commencing operation
krb5kdc: starting...
W20260927 18:38:27.442480 5727 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:27 dist-test-slave-19nl krb5kdc[27790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:27.555729 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41867
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:27.633767 27803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:27.633958 27803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:27.633978 27803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:27.635143 27803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:27.635182 27803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:27.635195 27803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:27.635241 27803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:27.635257 27803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:27.636569 27803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:27.636787 27803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:27.637017 27803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:27.641417 27808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:27.641454 27809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:27.641433 27811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:27.641984 27803 server_base.cc:1034] running on GCE node
I20260927 18:38:27.642206 27803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:27.642467 27803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:27.643603 27803 hybrid_clock.cc:648] HybridClock initialized: now 1790534307643605 us; error 47 us; skew 500 ppm
Sep 27 18:38:27 dist-test-slave-19nl krb5kdc[27790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:27.645887 27803 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:27.646294 27803 webserver.cc:460] Webserver started at http://127.5.151.254:41773/ using document root <none> and password file <none>
I20260927 18:38:27.646508 27803 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:27.646589 27803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:27.646760 27803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:27.647704 27803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "036d17052e64454e834df58236cb13b0"
format_stamp: "Formatted at 2026-09-27 18:38:27 on dist-test-slave-19nl"
server_key: "cba9eaf0b2171e9abcecbf7c7c8bea7e"
server_key_iv: "c747a567f993e4e0ea68366e37663a17"
server_key_version: "encryptionkey@0"
I20260927 18:38:27.648042 27803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "036d17052e64454e834df58236cb13b0"
format_stamp: "Formatted at 2026-09-27 18:38:27 on dist-test-slave-19nl"
server_key: "cba9eaf0b2171e9abcecbf7c7c8bea7e"
server_key_iv: "c747a567f993e4e0ea68366e37663a17"
server_key_version: "encryptionkey@0"
I20260927 18:38:27.649304 27803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:27.650089 27818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:27.650236 27803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:27.650334 27803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "036d17052e64454e834df58236cb13b0"
format_stamp: "Formatted at 2026-09-27 18:38:27 on dist-test-slave-19nl"
server_key: "cba9eaf0b2171e9abcecbf7c7c8bea7e"
server_key_iv: "c747a567f993e4e0ea68366e37663a17"
server_key_version: "encryptionkey@0"
I20260927 18:38:27.650408 27803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:27.653400 27803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:27.655614 27803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:27.655763 27803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:27.659385 27803 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:41867
I20260927 18:38:27.659415 27870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:41867 every 8 connection(s)
I20260927 18:38:27.659710 27803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:27.659950 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27803
I20260927 18:38:27.660063 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:27.660279 5727 external_mini_cluster.cc:1428] Setting key e183c0da983d34b096c6955656a1c054
I20260927 18:38:27.660813 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:27.662679 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:27.660814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38155 (local address 127.5.151.254:41867)
0927 18:38:27.660975 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:27.660983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:27.661010 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:38:27.661482 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:27.661490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:27.661521 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:27.661670 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:27.662249 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:27.662252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:27.662252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:27.662256 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:27.662297 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:27.662310 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:27.662382 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:27.662437 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:38:27.662474 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260927 18:38:27.663619 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:27.663885 27871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0: Bootstrap starting.
I20260927 18:38:27.664561 27871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:27.664901 27871 log.cc:824] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:27.665609 27871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0: No bootstrap required, opened a new log
I20260927 18:38:27.666281 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:27.665303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51577 (local address 127.5.151.254:41867)
0927 18:38:27.665338 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:27.665340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:27.665463 (+ 123us) server_negotiation.cc:407] Connection header received
0927 18:38:27.665499 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:27.665500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:27.665504 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:27.665567 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:27.666099 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:27.666100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:27.666101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:27.666105 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:27.666117 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:27.666121 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:27.666184 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:27.666218 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 18:38:27.666231 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:27.667577 27871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036d17052e64454e834df58236cb13b0" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41867 } }
I20260927 18:38:27.667719 27871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:27.667763 27871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036d17052e64454e834df58236cb13b0, State: Initialized, Role: FOLLOWER
I20260927 18:38:27.667876 27871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036d17052e64454e834df58236cb13b0" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41867 } }
I20260927 18:38:27.667961 27871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:27.668004 27871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:27.668054 27871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:27.668520 27871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036d17052e64454e834df58236cb13b0" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41867 } }
I20260927 18:38:27.668632 27871 leader_election.cc:302] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 036d17052e64454e834df58236cb13b0; no voters:
I20260927 18:38:27.668823 27871 leader_election.cc:288] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:27.668849 27881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:27.668962 27881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [term 1 LEADER]: Becoming Leader. State: Replica: 036d17052e64454e834df58236cb13b0, State: Running, Role: LEADER
I20260927 18:38:27.669095 27871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:27.669130 27881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036d17052e64454e834df58236cb13b0" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41867 } }
I20260927 18:38:27.669484 27882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "036d17052e64454e834df58236cb13b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036d17052e64454e834df58236cb13b0" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41867 } } }
I20260927 18:38:27.669497 27883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036d17052e64454e834df58236cb13b0. Latest consensus state: current_term: 1 leader_uuid: "036d17052e64454e834df58236cb13b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036d17052e64454e834df58236cb13b0" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41867 } } }
I20260927 18:38:27.669670 27883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:27.669655 27882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:27.670178 27890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:27.670766 27890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:27.672000 27890 catalog_manager.cc:1347] Generated new cluster ID: 8638a93336dd46d1b8bfeabfbc9aac3e
I20260927 18:38:27.672055 27890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:27.677160 27890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:27.677294 27890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:27.684145 27890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 036d17052e64454e834df58236cb13b0: Generated new TSK 0
I20260927 18:38:27.684304 27890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:27.691882 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27803
2026-09-27T18:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1289 ms)
[----------] 18 tests from AuthTokenIssuingTest (22346 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:27 dist-test-slave-19nl krb5kdc[27902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:27 dist-test-slave-19nl krb5kdc[27902](info): set up 2 sockets
Sep 27 18:38:27 dist-test-slave-19nl krb5kdc[27902](info): commencing operation
krb5kdc: starting...
W20260927 18:38:28.670702 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:28 dist-test-slave-19nl krb5kdc[27902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:28.775107 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41777
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41777,127.5.151.253:40057,127.5.151.252:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:28.849888 27915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:38:28.850088 27915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:28.850106 27915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:28.850124 27915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:28.851174 27915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:28.851207 27915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:28.851220 27915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:28.851233 27915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:28.851245 27915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:28.852591 27915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41777,127.5.151.253:40057,127.5.151.252:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:41777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:28.852766 27915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:28.852916 27915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:28.857601 27921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:28.857614 27923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:28.857610 27920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:28.857685 27915 server_base.cc:1034] running on GCE node
I20260927 18:38:28.857973 27915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:28.858274 27915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:28.859436 27915 hybrid_clock.cc:648] HybridClock initialized: now 1790534308859426 us; error 29 us; skew 500 ppm
Sep 27 18:38:28 dist-test-slave-19nl krb5kdc[27902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:28.861719 27915 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:28.862154 27915 webserver.cc:460] Webserver started at http://127.5.151.254:33075/ using document root <none> and password file <none>
I20260927 18:38:28.862371 27915 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:28.862435 27915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:28.862622 27915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:28.863567 27915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "21486a21ebde4c17b5e56bb9e905cdd5"
format_stamp: "Formatted at 2026-09-27 18:38:28 on dist-test-slave-19nl"
server_key: "d836cef128a37292e6c26c5af69fe5e3"
server_key_iv: "10a05b5af3566309d50b2976cafeb7c1"
server_key_version: "encryptionkey@0"
I20260927 18:38:28.863910 27915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "21486a21ebde4c17b5e56bb9e905cdd5"
format_stamp: "Formatted at 2026-09-27 18:38:28 on dist-test-slave-19nl"
server_key: "d836cef128a37292e6c26c5af69fe5e3"
server_key_iv: "10a05b5af3566309d50b2976cafeb7c1"
server_key_version: "encryptionkey@0"
I20260927 18:38:28.865185 27915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:28.865859 27930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:28.866024 27915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:28.866122 27915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "21486a21ebde4c17b5e56bb9e905cdd5"
format_stamp: "Formatted at 2026-09-27 18:38:28 on dist-test-slave-19nl"
server_key: "d836cef128a37292e6c26c5af69fe5e3"
server_key_iv: "10a05b5af3566309d50b2976cafeb7c1"
server_key_version: "encryptionkey@0"
I20260927 18:38:28.866204 27915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:28.879391 27915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:28.881105 27915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:28.881263 27915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:28.885205 27915 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:41777
I20260927 18:38:28.885234 27982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:41777 every 8 connection(s)
I20260927 18:38:28.885557 27915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:28.886479 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:28.888005 27983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:28.888861 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27915
I20260927 18:38:28.888998 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:28.889204 5727 external_mini_cluster.cc:1428] Setting key f21ce4db028958b8cce84670dcb5cfc9
Sep 27 18:38:28 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.151.253@KRBTEST.COM; defaulting to no policy
I20260927 18:38:28.898418 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:28.888621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41777 (local address 127.0.0.1:46036)
0927 18:38:28.888795 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:28.888833 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:28.889397 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:28.889700 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:28.889703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:28.889928 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:28.890091 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:28.890095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:28.891299 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:28.891301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:28.892111 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:28.892114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:28.892855 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:28.892856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:28.892908 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:28.893404 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:28.893419 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:28.895089 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:28.896938 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:28.896945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:28.896946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:28.897135 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:28.897412 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:28.897412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:28.897412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:28.897497 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:28.897970 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:28.897976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:28.898032 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 18:38:28.898117 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:38:28.898313 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":82,"thread_start_us":101,"threads_started":1}
I20260927 18:38:28.898451 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:28.888671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.5.151.254:41777)
0927 18:38:28.888830 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:28.888841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:28.888949 (+ 108us) server_negotiation.cc:407] Connection header received
0927 18:38:28.889561 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:28.889563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:28.889584 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:28.889685 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:28.890191 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:28.891209 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:28.892207 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:28.892757 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:28.892881 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:28.895187 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:28.895194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:28.895195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:28.895200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:28.896846 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:28.897253 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:28.897256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:28.897256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:28.897341 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:28.897562 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:28.897563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:28.897563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:28.897887 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:28.897961 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:28.898144 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 18:38:28.898238 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.253@KRBTEST.COM" created.
I20260927 18:38:28.899199 27983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:28.899619 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:28.899483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47190)
0927 18:38:28.899537 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:28.899591 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 18:38:28.899775 27932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
I20260927 18:38:28.900223 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:28.900110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47206)
0927 18:38:28.900159 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:28.900191 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 18:38:28.900401 27983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:40057: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab.
Entry for principal kudu/127.5.151.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab.
I20260927 18:38:28.912726 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:40057
--webserver_interface=127.5.151.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab
--principal=kudu/127.5.151.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41777,127.5.151.253:40057,127.5.151.252:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:28.949649 27983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } attempt: 1
I20260927 18:38:28.950157 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:28.950008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47218)
0927 18:38:28.950075 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:28.950122 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:38:28.950562 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:28.950437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47228)
0927 18:38:28.950481 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:28.950518 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 18:38:28.950660 27983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:40057: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
W20260927 18:38:28.990695 27990 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:38:28.990876 27990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:28.990892 27990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:28.990942 27990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:28.992070 27990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:28.992106 27990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:28.992120 27990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:28.992157 27990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:28.992183 27990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:28.993639 27990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41777,127.5.151.253:40057,127.5.151.252:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab
--principal=kudu/127.5.151.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.151.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:28.993889 27990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:28.994115 27990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:38:28.998791 27990 server_base.cc:1034] running on GCE node
W20260927 18:38:28.998838 27998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:28.998797 27996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:28.998808 27995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:28.999260 27990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:28.999490 27990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:29.000620 27990 hybrid_clock.cc:648] HybridClock initialized: now 1790534309000610 us; error 26 us; skew 500 ppm
Sep 27 18:38:28 dist-test-slave-19nl krb5kdc[27902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:29.002982 27990 init.cc:377] Logged in from keytab as kudu/127.5.151.253@KRBTEST.COM (short username kudu)
I20260927 18:38:29.003403 27990 webserver.cc:460] Webserver started at http://127.5.151.253:39661/ using document root <none> and password file <none>
I20260927 18:38:29.003636 27990 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:29.003679 27990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:29.003827 27990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:29.004839 27990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data/instance:
uuid: "5456db7c36e8410fb7cdad6cee27a681"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-19nl"
server_key: "43fa7997f51d9cf5315fe6675e742467"
server_key_iv: "63e54f078165d387716a842a22bc10ba"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.005183 27990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/wal/instance:
uuid: "5456db7c36e8410fb7cdad6cee27a681"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-19nl"
server_key: "43fa7997f51d9cf5315fe6675e742467"
server_key_iv: "63e54f078165d387716a842a22bc10ba"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.006410 27990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 18:38:29.007159 28005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:29.007323 27990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:29.007418 27990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/wal
uuid: "5456db7c36e8410fb7cdad6cee27a681"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-19nl"
server_key: "43fa7997f51d9cf5315fe6675e742467"
server_key_iv: "63e54f078165d387716a842a22bc10ba"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.007496 27990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:29.018823 27983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } attempt: 2
I20260927 18:38:29.019222 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.019104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47230)
0927 18:38:29.019160 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.019198 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 18:38:29.019595 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.019495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47236)
0927 18:38:29.019545 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.019570 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:38:29.019675 27983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:40057: Network error: Client connection negotiation failed: client connection to 127.5.151.253:40057: connect: Connection refused (error 111)
I20260927 18:38:29.023484 27990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:29.025373 27990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:29.025525 27990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:29.029543 27990 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.253:40057
I20260927 18:38:29.029574 28057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.253:40057 every 8 connection(s)
I20260927 18:38:29.029933 27990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
I20260927 18:38:29.030977 28058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:29.032493 28058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:29.037092 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 27990
I20260927 18:38:29.037169 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-1/wal/instance
I20260927 18:38:29.037334 5727 external_mini_cluster.cc:1428] Setting key 69d053bddf37b6df1b75cc4d745e0e4d
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:29.041554 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.033258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.5.151.254:41777)
0927 18:38:29.033318 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.033320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.033542 (+ 222us) server_negotiation.cc:407] Connection header received
0927 18:38:29.033995 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.033996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.034005 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.034073 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.034711 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.035167 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.036079 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.036261 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.036310 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.038492 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.038494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.038495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.038503 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.040213 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.040581 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.040582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.040583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.040646 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.040872 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.040873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.040874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.041026 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.041109 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.041346 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.041433 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:38:29.041561 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.033229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41777 (local address 127.0.0.1:46048)
0927 18:38:29.033417 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.033457 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.033915 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.034133 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.034137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.034384 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.034638 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.034643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.035242 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.035243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.035994 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.035996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.036343 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.036344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.036441 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.036954 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.036962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.038360 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.040291 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.040297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.040299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.040507 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.040712 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.040712 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.040713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.040806 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.041103 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.041108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.041218 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.041299 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.041410 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260927 18:38:29.042297 28058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.151.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.252@KRBTEST.COM" created.
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:38:29.050398 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.042538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47250)
0927 18:38:29.042613 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.042622 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.042656 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.042949 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.042949 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.043079 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.043121 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.043123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.043910 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.043911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.044390 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.044392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.044905 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.044906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.044969 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.045354 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.045359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.047626 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.049428 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.049429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.049429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.049583 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.049790 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.049791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.049791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.049837 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.050143 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.050144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.050204 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.050278 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.050343 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:38:29.050570 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.042667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.5.151.253:40057)
0927 18:38:29.042797 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.042801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.042833 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:38:29.042857 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.042859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.042868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.042948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.043195 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.043827 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.044483 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.044826 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.044897 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.047717 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.047724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.047724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.047732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.049349 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.049662 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.049665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.049665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.049726 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.049892 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.049893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.049893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.050077 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.050164 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.050334 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.050424 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260927 18:38:29.051123 28058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:29.051438 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.051333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43510)
0927 18:38:29.051383 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.051422 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:38:29.051625 28009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
I20260927 18:38:29.052052 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.051956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43514)
0927 18:38:29.052006 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.052033 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 18:38:29.052181 28058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:33901: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab.
Entry for principal kudu/127.5.151.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab.
I20260927 18:38:29.061476 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:33901
--webserver_interface=127.5.151.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab
--principal=kudu/127.5.151.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:41777,127.5.151.253:40057,127.5.151.252:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:29.101395 28058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } attempt: 1
I20260927 18:38:29.101814 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.101692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43520)
0927 18:38:29.101753 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.101786 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 18:38:29.102123 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.102031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43534)
0927 18:38:29.102075 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.102098 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:38:29.102198 28058 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:33901: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
I20260927 18:38:29.110805 27983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } attempt: 3
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:38:29.117781 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.111165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.5.151.253:40057)
0927 18:38:29.111213 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.111215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.111232 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:38:29.111274 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.111275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.111283 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.111330 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.111685 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.112146 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.113137 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.113339 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.113383 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.115287 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.115288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.115289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.115295 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.116679 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.117045 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.117046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.117046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.117101 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.117293 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.117294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.117294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.117401 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.117475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.117628 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.117697 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:38:29.117787 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.111099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47264)
0927 18:38:29.111143 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.111157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.111218 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.111347 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.111348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.111533 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.111617 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.111618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.112213 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.112214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.113057 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.113060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.113405 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.113406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.113483 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.113889 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.113894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.115205 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.116779 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.116780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.116780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.116955 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.117167 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.117168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.117168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.117231 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.117467 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.117468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.117514 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.117603 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.117697 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 18:38:29.118230 27983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:29.118537 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.118442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43538)
0927 18:38:29.118486 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.118513 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:38:29.118937 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.118839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43542)
0927 18:38:29.118901 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.118924 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:38:29.119037 27983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:33901: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
W20260927 18:38:29.139137 28065 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:38:29.139312 28065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:29.139328 28065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:29.139390 28065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:29.140501 28065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:29.140537 28065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:29.140551 28065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:29.140564 28065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:29.140612 28065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:29.142009 28065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:41777,127.5.151.253:40057,127.5.151.252:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab
--principal=kudu/127.5.151.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.151.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:29.142262 28065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:29.142474 28065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:29.146915 28073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:29.147063 28065 server_base.cc:1034] running on GCE node
W20260927 18:38:29.146951 28070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:29.146929 28071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:29.147400 28065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:29.147667 28065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:29.148821 28065 hybrid_clock.cc:648] HybridClock initialized: now 1790534309148813 us; error 34 us; skew 500 ppm
I20260927 18:38:29.150162 27983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } attempt: 1
I20260927 18:38:29.150480 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.150372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43554)
0927 18:38:29.150422 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.150450 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 18:38:29.150839 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.150743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43560)
0927 18:38:29.150790 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.150818 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:38:29.150940 27983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:33901: Network error: Client connection negotiation failed: client connection to 127.5.151.252:33901: connect: Connection refused (error 111)
I20260927 18:38:29.151201 28065 init.cc:377] Logged in from keytab as kudu/127.5.151.252@KRBTEST.COM (short username kudu)
I20260927 18:38:29.151542 28065 webserver.cc:460] Webserver started at http://127.5.151.252:38455/ using document root <none> and password file <none>
I20260927 18:38:29.151700 28065 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:29.151736 28065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:29.151814 28065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:29.152725 28065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data/instance:
uuid: "3ae951a36c8d4be2ada3280155db87c2"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-19nl"
server_key: "f36c3e3f4d981d205188f9131a9dc6f1"
server_key_iv: "7ba71c0d0dc0d995f2bd7158cfbdcb05"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.153015 28065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/wal/instance:
uuid: "3ae951a36c8d4be2ada3280155db87c2"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-19nl"
server_key: "f36c3e3f4d981d205188f9131a9dc6f1"
server_key_iv: "7ba71c0d0dc0d995f2bd7158cfbdcb05"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.154191 28065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:38:29.154843 28080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:29.155015 28065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:29.155097 28065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/wal
uuid: "3ae951a36c8d4be2ada3280155db87c2"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-19nl"
server_key: "f36c3e3f4d981d205188f9131a9dc6f1"
server_key_iv: "7ba71c0d0dc0d995f2bd7158cfbdcb05"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.155164 28065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:29.164781 28065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:29.166311 28065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:29.166469 28065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:29.170215 28065 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.252:33901
I20260927 18:38:29.170249 28132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.252:33901 every 8 connection(s)
I20260927 18:38:29.170306 28058 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } attempt: 2
I20260927 18:38:29.170713 28065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
I20260927 18:38:29.171482 28133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:29.173193 28133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:29.175475 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 28065
I20260927 18:38:29.175607 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534055081918-5727-0/minicluster-data/master-2/wal/instance
I20260927 18:38:29.175792 5727 external_mini_cluster.cc:1428] Setting key d946141567b2370a7ba2d33930b7ecdb
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:29.181325 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.170632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43568)
0927 18:38:29.170678 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.170688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.170730 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.172138 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.172140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.172303 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.172354 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.172356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.173390 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.173391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.174073 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.174075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.174643 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.174644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.174700 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.175133 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.175138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.176677 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.179100 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.179101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.179102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.179297 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.180505 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.180506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.180506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.180568 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.180996 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.180997 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.181055 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.181135 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.181221 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 18:38:29.181547 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.170889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.5.151.252:33901)
0927 18:38:29.171508 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.171514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.171537 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:38:29.171958 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.171963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.172010 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.172115 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.172449 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.173298 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.174142 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.174508 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.174673 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.176775 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.176782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.176782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.176793 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.179001 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.179631 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.179634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.179634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.179752 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.180644 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.180645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.180645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.180913 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.180998 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.181199 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.181318 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":541,"threads_started":1}
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:38:29.181916 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.173779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.5.151.254:41777)
0927 18:38:29.173834 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.173836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.174269 (+ 433us) server_negotiation.cc:407] Connection header received
0927 18:38:29.174333 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.174335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.174345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.174399 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.174884 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.175307 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.176352 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.176654 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.176736 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.178908 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.178909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.178910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.178916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.180406 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.180821 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.180822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.180823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.180871 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.181052 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.181053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.181053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.181173 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.181232 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.181751 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.181828 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:38:29.182209 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.173713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:41777 (local address 127.0.0.1:46058)
0927 18:38:29.174182 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.174211 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.174283 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.174424 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.174426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.174737 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.174798 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.174800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.175394 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.175395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.176270 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.176272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.176763 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.176764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.176831 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.177293 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.177298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.178825 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.180487 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.180493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.180495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.180739 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.180940 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.180940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.180941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.180995 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.181256 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.181260 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.181302 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.181994 (+ 692us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.182111 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":91,"threads_started":1}
I20260927 18:38:29.182777 28133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:29.183280 28058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: Bootstrap starting.
I20260927 18:38:29.184046 28058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:29.184351 28058 log.cc:824] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:29.185235 28058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: No bootstrap required, opened a new log
I20260927 18:38:29.185326 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.176120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.5.151.252:33901)
0927 18:38:29.176228 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.176230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.176384 (+ 154us) server_negotiation.cc:407] Connection header received
0927 18:38:29.176453 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.176454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.176463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.176529 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.177303 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.178042 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.179596 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.179800 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.179867 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.182081 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.182082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.182082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.182088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.184140 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.184521 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.184522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.184522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.184570 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.184779 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.184781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.184781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.184905 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.185037 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.185154 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.185225 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260927 18:38:29.187443 28058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.187584 28058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5456db7c36e8410fb7cdad6cee27a681, State: Initialized, Role: FOLLOWER
I20260927 18:38:29.187698 28058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.188030 28058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:29.188000 28140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.188186 28140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
W20260927 18:38:29.189848 28151 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:29.189939 28151 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:29.191967 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.184678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:40057 (local address 127.0.0.1:47270)
0927 18:38:29.184709 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.184718 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.184748 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.184897 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.184897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.185056 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.185128 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.185129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.185924 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.185925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.186374 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.186375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.186856 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.186857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.186923 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.187336 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.187340 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.188558 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.190581 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.190582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.190583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.190768 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.190990 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.190991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.190992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.191052 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.191362 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.191363 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.191405 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.191771 (+ 366us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.191862 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 18:38:29.190124 27990 master_runner.cc:186] Error getting master registration for 127.5.151.254:41777: OK, instance_id { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" instance_seqno: 1790534308884116 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:38:29.192827 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.184759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.5.151.253:40057)
0927 18:38:29.184792 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.184793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.184800 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:38:29.184817 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.184818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.184827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.184876 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.185220 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.185836 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.186454 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.186795 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.186858 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.188675 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.188676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.188676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.188683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.190458 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.190867 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.190868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.190869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.190923 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.191124 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.191124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.191125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.191276 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.191335 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.192654 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.192724 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:29.192878 27990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:38:29.193243 28133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:29.194168 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.185891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.5.151.254:41777)
0927 18:38:29.185922 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.185925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.186041 (+ 116us) server_negotiation.cc:407] Connection header received
0927 18:38:29.186059 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.186059 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.186067 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.186111 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.186711 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.187418 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.188349 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.188651 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.188912 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.191302 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.191303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.191304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.191310 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.193030 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.193435 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.193436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.193436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.193501 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.193726 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.193727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.193727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.193841 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.193981 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.194063 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.194117 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:38:29.201293 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.193470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43584)
0927 18:38:29.193523 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.193533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.193566 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.193682 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.193683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.193816 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.193845 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.193846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.195210 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.195210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.195917 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.195919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.196414 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.196415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.196459 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.197022 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.197026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.198127 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.200355 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.200356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.200356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.200505 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.200706 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.200707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.200707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.200757 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.201010 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.201011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.201045 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.201109 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.201196 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:38:29.201350 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.193558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.5.151.252:33901)
0927 18:38:29.193590 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.193592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.193601 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:29.193620 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.193620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.193628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.193680 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.194227 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.195125 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.196002 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.196297 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.196368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.198214 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.198215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.198215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.198221 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.200080 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.200590 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.200591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.200591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.200638 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.200824 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.200825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.200825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.200942 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.201017 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.201137 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.201244 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":17}
I20260927 18:38:29.202293 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.194827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.5.151.253:40057)
0927 18:38:29.194862 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.194863 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.194871 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:38:29.194889 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.194890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.194897 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.194960 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.195421 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.195833 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.196462 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.196751 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.196813 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.199476 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.199477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.199477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.199483 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.200934 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.201320 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.201321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.201322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.201601 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.201815 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.201816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.201816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.201925 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.202050 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.202143 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.202210 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 18:38:29.202854 28133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: Bootstrap starting.
I20260927 18:38:29.203547 28133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:29.203675 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:29.203845 28133 log.cc:824] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:29.204512 28133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: No bootstrap required, opened a new log
I20260927 18:38:29.204599 27957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "21486a21ebde4c17b5e56bb9e905cdd5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46070
I20260927 18:38:29.206350 28133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.206491 28133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae951a36c8d4be2ada3280155db87c2, State: Initialized, Role: FOLLOWER
I20260927 18:38:29.206589 28133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.206858 28133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:29.206830 28154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.206916 28154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:38:29.209384 28165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:29.209441 28165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:29.208748 28065 master_runner.cc:186] Error getting master registration for 127.5.151.254:41777: OK, instance_id { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" instance_seqno: 1790534308884116 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:38:29.211542 28065 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:38:29.226065 27983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } attempt: 2
Sep 27 18:38:29 dist-test-slave-19nl krb5kdc[27902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:38:29.233735 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.226371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:33901 (local address 127.0.0.1:43586)
0927 18:38:29.226421 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:29.226432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:29.226468 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:29.226611 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:29.226611 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:29.226800 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:29.226848 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.226850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.227719 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.227721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.228514 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:29.228516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.228978 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.228979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:29.229037 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.229472 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:29.229477 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:29.230941 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:29.232715 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.232716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.232717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.232927 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.233156 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:29.233157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:29.233157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:29.233210 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:29.233457 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:29.233458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:29.233498 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:38:29.233556 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:38:29.233640 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 18:38:29.234071 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.226426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.5.151.252:33901)
0927 18:38:29.226462 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.226464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.226474 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:38:29.226518 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.226519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.226528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.226606 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.226929 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.227629 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.228587 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.228885 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.228975 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.231049 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.231050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.231051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.231058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.232627 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.233039 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.233040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.233040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.233092 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.233271 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.233272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.233272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.233381 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.233435 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.233884 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.233973 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:38:29.235642 27983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5: Bootstrap starting.
I20260927 18:38:29.236299 27983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:29.236591 27983 log.cc:824] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:29.237505 27983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5: No bootstrap required, opened a new log
I20260927 18:38:29.238811 27983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.238940 27983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21486a21ebde4c17b5e56bb9e905cdd5, State: Initialized, Role: FOLLOWER
I20260927 18:38:29.239024 27983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.239372 27957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:38:29.239442 27957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:29.240152 27957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.240399 27957 leader_election.cc:288] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [CANDIDATE]: Term 1 election: Requested vote from peers 5456db7c36e8410fb7cdad6cee27a681 (127.5.151.253:40057), 3ae951a36c8d4be2ada3280155db87c2 (127.5.151.252:33901)
I20260927 18:38:29.239315 27983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:29.240796 28032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5456db7c36e8410fb7cdad6cee27a681"
I20260927 18:38:29.240919 28032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:29.239308 28168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.241680 28168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:29.241784 28032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21486a21ebde4c17b5e56bb9e905cdd5 in term 1.
I20260927 18:38:29.242786 27932 leader_election.cc:302] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21486a21ebde4c17b5e56bb9e905cdd5, 5456db7c36e8410fb7cdad6cee27a681; no voters:
I20260927 18:38:29.242988 28168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:29.243146 28168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [term 1 LEADER]: Becoming Leader. State: Replica: 21486a21ebde4c17b5e56bb9e905cdd5, State: Running, Role: LEADER
I20260927 18:38:29.243232 28168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } }
I20260927 18:38:29.243616 28168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21486a21ebde4c17b5e56bb9e905cdd5. Latest consensus state: current_term: 1 leader_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.243690 28168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:29.243912 28107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ae951a36c8d4be2ada3280155db87c2"
I20260927 18:38:29.244020 28107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:38:29.244315 28179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:29.244379 28179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:29.244850 28107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21486a21ebde4c17b5e56bb9e905cdd5 in term 1.
I20260927 18:38:29.247074 28187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:29.247249 28187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:29.254222 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.245877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.5.151.254:41777)
0927 18:38:29.245912 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.245914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.246986 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 18:38:29.247097 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.247098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.247108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.247142 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.247824 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.248526 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.249499 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.249767 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.249812 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.251391 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.251392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.251393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.251399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.253023 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.253396 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.253397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.253398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.253453 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.253645 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.253646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.253647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.253761 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.253826 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.254047 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.254120 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:38:29.255317 28107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [term 1 FOLLOWER]: Refusing update from remote peer 21486a21ebde4c17b5e56bb9e905cdd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:29.255401 28032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [term 1 FOLLOWER]: Refusing update from remote peer 21486a21ebde4c17b5e56bb9e905cdd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:29.255815 28184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:29.256029 28192 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:29.257244 28140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21486a21ebde4c17b5e56bb9e905cdd5. Latest consensus state: current_term: 1 leader_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.257351 28140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:29.257759 28154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21486a21ebde4c17b5e56bb9e905cdd5. Latest consensus state: current_term: 1 leader_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.257858 28154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:29.261591 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.246191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.5.151.252:33901)
0927 18:38:29.246224 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.246226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.247534 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 18:38:29.247629 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.247630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.247639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.247668 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.250068 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.250765 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.251718 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.251994 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.252029 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.255525 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.255526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.255527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.255533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.257452 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.260222 (+ 2770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.260223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.260223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.260275 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.260700 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.260701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.260701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.260817 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.260891 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.261440 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.261509 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:29.262151 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.243320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.5.151.253:40057)
0927 18:38:29.243372 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.243374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.246443 (+ 3069us) server_negotiation.cc:407] Connection header received
0927 18:38:29.246593 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.246594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.246607 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.246638 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.248387 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.249062 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.252107 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.252389 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.252425 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.258243 (+ 5818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.258244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.258245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.258252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.259814 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.260462 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.260463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.260463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.260510 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.261067 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.261068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.261068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.261187 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.261247 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.261989 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.262069 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:38:29.263450 28191 mvcc.cc:205] Tried to move back new op lower bound from 7334028530707730432 to 7334028530660954112. Current Snapshot: MvccSnapshot[applied={T|T < 7334028530707730432}]
I20260927 18:38:29.264084 28140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.264166 28140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:29.263634 28184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.264341 28184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:29.267066 28187 catalog_manager.cc:1347] Generated new cluster ID: ffa561b008e24efd99425befeecfffeb
I20260927 18:38:29.267123 28187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:29.267429 28192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.267506 28192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:29.267795 28154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21486a21ebde4c17b5e56bb9e905cdd5" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 41777 } } peers { permanent_uuid: "5456db7c36e8410fb7cdad6cee27a681" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 40057 } } peers { permanent_uuid: "3ae951a36c8d4be2ada3280155db87c2" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 33901 } } }
I20260927 18:38:29.267890 28154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:29.276602 28187 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:29.277227 28187 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:29.287510 28187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21486a21ebde4c17b5e56bb9e905cdd5: Generated new TSK 0
I20260927 18:38:29.287709 28187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:38:29.312706 27986 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.311180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.5.151.254:41777)
0927 18:38:29.311219 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.311221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.312080 (+ 859us) server_negotiation.cc:407] Connection header received
0927 18:38:29.312180 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.312181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.312192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.312221 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.312642 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46082: BlockingRecv error: recv got EOF from 127.0.0.1:46082 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 18:38:29.313416 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.311379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.5.151.253:40057)
0927 18:38:29.311416 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.311418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.312925 (+ 1507us) server_negotiation.cc:407] Connection header received
0927 18:38:29.313041 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.313042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.313051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.313080 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.313363 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47290: BlockingRecv error: recv got EOF from 127.0.0.1:47290 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 18:38:29.314091 28137 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.311819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.5.151.252:33901)
0927 18:38:29.311855 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.311857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.313630 (+ 1773us) server_negotiation.cc:407] Connection header received
0927 18:38:29.313719 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.313720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.313729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.313757 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.314038 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43614: BlockingRecv error: recv got EOF from 127.0.0.1:43614 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 18:38:29.316365 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.315591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.5.151.253:40057)
0927 18:38:29.315623 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.315625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.315874 (+ 249us) server_negotiation.cc:407] Connection header received
0927 18:38:29.315979 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.315980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.316000 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.316029 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.316323 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47296: BlockingRecv error: recv got EOF from 127.0.0.1:47296 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 18:38:29.319341 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.318576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.5.151.253:40057)
0927 18:38:29.318606 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.318608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.318852 (+ 244us) server_negotiation.cc:407] Connection header received
0927 18:38:29.318943 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.318944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.318952 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.318980 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.319304 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47304: BlockingRecv error: recv got EOF from 127.0.0.1:47304 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 18:38:29.323179 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.322395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.5.151.253:40057)
0927 18:38:29.322427 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.322429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.322716 (+ 287us) server_negotiation.cc:407] Connection header received
0927 18:38:29.322809 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.322810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.322820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.322848 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.323141 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47308: BlockingRecv error: recv got EOF from 127.0.0.1:47308 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 18:38:29.329366 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.328680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.5.151.253:40057)
0927 18:38:29.328713 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.328715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.328937 (+ 222us) server_negotiation.cc:407] Connection header received
0927 18:38:29.329018 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.329019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.329030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.329058 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.329328 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47314: BlockingRecv error: recv got EOF from 127.0.0.1:47314 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 18:38:29.339215 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.338431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.5.151.253:40057)
0927 18:38:29.338479 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.338481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.338756 (+ 275us) server_negotiation.cc:407] Connection header received
0927 18:38:29.338845 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.338846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.338855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.338881 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.339177 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47324: BlockingRecv error: recv got EOF from 127.0.0.1:47324 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 18:38:29.357331 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.356422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.5.151.253:40057)
0927 18:38:29.356460 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.356462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.356718 (+ 256us) server_negotiation.cc:407] Connection header received
0927 18:38:29.356812 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.356813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.356824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.356852 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.357244 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47330: BlockingRecv error: recv got EOF from 127.0.0.1:47330 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 18:38:29.391357 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.390629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.5.151.253:40057)
0927 18:38:29.390677 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.390679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.390819 (+ 140us) server_negotiation.cc:407] Connection header received
0927 18:38:29.390894 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.390895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.390906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.390978 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.391289 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47344: BlockingRecv error: recv got EOF from 127.0.0.1:47344 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 18:38:29.457165 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.456443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.5.151.253:40057)
0927 18:38:29.456649 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.456651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.456660 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:29.456686 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.456687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.456697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.456747 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.457075 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47358: BlockingRecv error: recv got EOF from 127.0.0.1:47358 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
W20260927 18:38:29.587018 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.586312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.5.151.253:40057)
0927 18:38:29.586375 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.586377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.586490 (+ 113us) server_negotiation.cc:407] Connection header received
0927 18:38:29.586522 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.586524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.586536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.586676 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.586970 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47368: BlockingRecv error: recv got EOF from 127.0.0.1:47368 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 18:38:29.844798 28062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:29.844118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.5.151.253:40057)
0927 18:38:29.844184 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.844185 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.844271 (+ 86us) server_negotiation.cc:407] Connection header received
0927 18:38:29.844329 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.844330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.844340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.844418 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.844730 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47376: BlockingRecv error: recv got EOF from 127.0.0.1:47376 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:38:30.190395 28151 catalog_manager.cc:1259] Loaded cluster ID: ffa561b008e24efd99425befeecfffeb
I20260927 18:38:30.190462 28151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: loading cluster ID for follower catalog manager: success
I20260927 18:38:30.191305 28151 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: acquiring CA information for follower catalog manager: success
I20260927 18:38:30.191618 28151 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5456db7c36e8410fb7cdad6cee27a681: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:38:30.209853 28165 catalog_manager.cc:1259] Loaded cluster ID: ffa561b008e24efd99425befeecfffeb
I20260927 18:38:30.209913 28165 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: loading cluster ID for follower catalog manager: success
I20260927 18:38:30.210780 28165 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: acquiring CA information for follower catalog manager: success
I20260927 18:38:30.211099 28165 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ae951a36c8d4be2ada3280155db87c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:38:30.360718 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.358084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.5.151.253:40057)
0927 18:38:30.358246 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:30.358249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:30.358261 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:38:30.358345 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:30.358347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:30.358415 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:30.358501 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:30.358668 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.359153 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.359912 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.360109 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.360177 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.360371 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:30.360476 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:30.360569 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:30.360591 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 18:38:30.360611 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260927 18:38:30.365293 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.362603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.5.151.252:33901)
0927 18:38:30.362719 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:30.362721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:30.362733 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:38:30.362783 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:30.362784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:30.362843 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:30.362934 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:30.363035 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.363876 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.364623 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.364808 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.364860 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.364973 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:30.365068 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:30.365131 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:30.365162 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 18:38:30.365179 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260927 18:38:30.369386 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.366675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.5.151.253:40057)
0927 18:38:30.366734 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:30.366753 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:30.366927 (+ 174us) server_negotiation.cc:407] Connection header received
0927 18:38:30.367014 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:30.367015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:30.367025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:30.367080 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:30.367225 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.367732 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.368563 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.368868 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.368930 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.369122 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:30.369151 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:30.369220 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:30.369278 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 18:38:30.369297 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:38:30.369863 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.366926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.5.151.252:33901)
0927 18:38:30.366966 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:30.366967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:30.367120 (+ 153us) server_negotiation.cc:407] Connection header received
0927 18:38:30.367221 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:30.367222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:30.367233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:30.367291 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:30.367390 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.368050 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.369063 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.369329 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.369384 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.369586 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:30.369615 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:30.369640 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:30.369770 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 18:38:30.369791 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:38:30.370267 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.366813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.5.151.254:41777)
0927 18:38:30.366988 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:30.366990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:30.366998 (+ 8us) server_negotiation.cc:407] Connection header received
0927 18:38:30.367041 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:30.367041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:30.367089 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:30.367161 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:30.367298 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.368047 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.369081 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.369378 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.369431 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.369764 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:30.369889 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:30.369990 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:30.370142 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:38:30.370170 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260927 18:38:30.371687 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27915
I20260927 18:38:30.377229 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 27990
I20260927 18:38:30.381682 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 28065
2026-09-27T18:38:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2691 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:30 dist-test-slave-19nl krb5kdc[28281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:30 dist-test-slave-19nl krb5kdc[28281](info): set up 2 sockets
Sep 27 18:38:30 dist-test-slave-19nl krb5kdc[28281](info): commencing operation
krb5kdc: starting...
W20260927 18:38:31.382426 5727 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/test-user.keytab.
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.151.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
Entry for principal kudu/127.5.151.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab.
I20260927 18:38:31.485128 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32913
--webserver_interface=127.5.151.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32913,127.5.151.253:37795,127.5.151.252:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:31.562073 28294 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:38:31.562261 28294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:31.562278 28294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:31.562296 28294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:31.563402 28294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:31.563439 28294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:31.563452 28294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:31.563489 28294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:31.563519 28294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:31.564903 28294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32913,127.5.151.253:37795,127.5.151.252:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.254.keytab
--principal=kudu/127.5.151.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.254:32913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.151.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:31.565146 28294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:31.565366 28294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:38:31.570003 28294 server_base.cc:1034] running on GCE node
W20260927 18:38:31.569990 28302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.569968 28300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.569975 28299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:31.570436 28294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:31.570770 28294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:31.571935 28294 hybrid_clock.cc:648] HybridClock initialized: now 1790534311571922 us; error 42 us; skew 500 ppm
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:31.574663 28294 init.cc:377] Logged in from keytab as kudu/127.5.151.254@KRBTEST.COM (short username kudu)
I20260927 18:38:31.575057 28294 webserver.cc:460] Webserver started at http://127.5.151.254:39609/ using document root <none> and password file <none>
I20260927 18:38:31.575253 28294 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:31.575292 28294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:31.575378 28294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:31.576411 28294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data/instance:
uuid: "7a5e503f47aa421195a948c89693cf7c"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "cf0b7c4ca7a34b9637826f29523b2900"
server_key_iv: "7bccdc46c081e40e8038e3fbe51c6fbb"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.576762 28294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/wal/instance:
uuid: "7a5e503f47aa421195a948c89693cf7c"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "cf0b7c4ca7a34b9637826f29523b2900"
server_key_iv: "7bccdc46c081e40e8038e3fbe51c6fbb"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.578022 28294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 18:38:31.578800 28309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:31.578991 28294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:38:31.579087 28294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/wal
uuid: "7a5e503f47aa421195a948c89693cf7c"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "cf0b7c4ca7a34b9637826f29523b2900"
server_key_iv: "7bccdc46c081e40e8038e3fbe51c6fbb"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.579171 28294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:31.593816 28294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:31.595598 28294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:31.595749 28294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:31.599429 28294 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.254:32913
I20260927 18:38:31.599474 28361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.254:32913 every 8 connection(s)
I20260927 18:38:31.599807 28294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:31.600683 28362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:31.602101 28362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:31.609294 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 28294
I20260927 18:38:31.609368 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-0/wal/instance
I20260927 18:38:31.609566 5727 external_mini_cluster.cc:1428] Setting key e52156668d8961bc1da845037811032a
I20260927 18:38:31.611152 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.602827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.5.151.254:32913)
0927 18:38:31.602951 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.602955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.602999 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:38:31.603488 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.603491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.603509 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.603622 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.604130 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.604881 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.605728 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.606033 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.606153 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.607984 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.607991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.607991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.607997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.609549 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.609978 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.609980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.609981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.610058 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.610313 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.610313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.610314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.610488 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.610598 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.610811 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.610898 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":39,"threads_started":1}
I20260927 18:38:31.611159 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.602690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32913 (local address 127.0.0.1:41382)
0927 18:38:31.602824 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.602854 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.603338 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.603629 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.603631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.603891 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.604032 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.604036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.605004 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.605005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.605635 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.605636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.606140 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.606141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.606204 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.606638 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.606646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.607905 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.609626 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.609634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.609636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.609901 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.610127 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.610128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.610128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.610251 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.610573 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.610579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.610667 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.610777 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.611001 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":70,"thread_start_us":60,"threads_started":1}
I20260927 18:38:31.611864 28362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:31.612196 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.612071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40640)
0927 18:38:31.612120 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.612172 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:38:31.612334 28312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
I20260927 18:38:31.612766 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.612662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40644)
0927 18:38:31.612709 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.612738 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:38:31.612914 28362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:37795: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.151.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab.
Entry for principal kudu/127.5.151.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab.
I20260927 18:38:31.630873 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:37795
--webserver_interface=127.5.151.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab
--principal=kudu/127.5.151.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32913,127.5.151.253:37795,127.5.151.252:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:31.662155 28362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } attempt: 1
I20260927 18:38:31.662653 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.662493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40648)
0927 18:38:31.662580 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.662621 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:38:31.663018 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.662920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40662)
0927 18:38:31.662973 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.663000 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 18:38:31.663115 28362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:37795: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
W20260927 18:38:31.705083 28369 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:38:31.705250 28369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:31.705264 28369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:31.705282 28369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:31.706214 28369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:31.706244 28369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:31.706257 28369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:31.706269 28369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:31.706279 28369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:31.707592 28369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32913,127.5.151.253:37795,127.5.151.252:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.253.keytab
--principal=kudu/127.5.151.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.253:37795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.151.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:31.707751 28369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:31.707890 28369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:31.712075 28377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.712103 28375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.712105 28374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:31.712296 28369 server_base.cc:1034] running on GCE node
I20260927 18:38:31.712477 28369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:31.712718 28369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:31.713847 28369 hybrid_clock.cc:648] HybridClock initialized: now 1790534311713837 us; error 23 us; skew 500 ppm
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:31.716291 28369 init.cc:377] Logged in from keytab as kudu/127.5.151.253@KRBTEST.COM (short username kudu)
I20260927 18:38:31.716683 28369 webserver.cc:460] Webserver started at http://127.5.151.253:33167/ using document root <none> and password file <none>
I20260927 18:38:31.716898 28369 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:31.716962 28369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:31.717120 28369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:31.718048 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data/instance:
uuid: "196e6341474146f685102d52582cf2cf"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "869a0710270b0e85055aab4e9c597653"
server_key_iv: "fa419871fa033ded6808d2ccb8d1d8bb"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.718379 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/wal/instance:
uuid: "196e6341474146f685102d52582cf2cf"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "869a0710270b0e85055aab4e9c597653"
server_key_iv: "fa419871fa033ded6808d2ccb8d1d8bb"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.719664 28369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:38:31.720350 28384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:31.720520 28369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 18:38:31.720615 28369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/wal
uuid: "196e6341474146f685102d52582cf2cf"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "869a0710270b0e85055aab4e9c597653"
server_key_iv: "fa419871fa033ded6808d2ccb8d1d8bb"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.720697 28369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:31.731230 28362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } attempt: 2
I20260927 18:38:31.731592 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.731469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40674)
0927 18:38:31.731525 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.731560 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 18:38:31.731914 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.731828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40690)
0927 18:38:31.731869 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.731894 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 18:38:31.732059 28362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.253:37795: Network error: Client connection negotiation failed: client connection to 127.5.151.253:37795: connect: Connection refused (error 111)
I20260927 18:38:31.743942 28369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:31.745574 28369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:31.745718 28369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:31.749323 28369 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.253:37795
I20260927 18:38:31.749349 28436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.253:37795 every 8 connection(s)
I20260927 18:38:31.749640 28369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/data/info.pb
I20260927 18:38:31.750478 28437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:31.751847 28437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:31.754881 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 28369
I20260927 18:38:31.754976 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-1/wal/instance
I20260927 18:38:31.755146 5727 external_mini_cluster.cc:1428] Setting key acb02d3a0d2124af2f708164b6735c79
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:31.760565 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.752429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.5.151.254:32913)
0927 18:38:31.752485 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.752487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.752687 (+ 200us) server_negotiation.cc:407] Connection header received
0927 18:38:31.753111 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.753112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.753121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.753171 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.753659 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.754102 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.755288 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.755476 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.755529 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.757763 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.757764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.757764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.757770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.759219 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.759701 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.759702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.759702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.759748 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.760015 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.760015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.760015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.760108 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.760153 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.760446 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.760505 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:38:31.760804 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.752423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32913 (local address 127.0.0.1:41388)
0927 18:38:31.752572 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.752608 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.753033 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.753216 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.753219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.753421 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.753595 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.753598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.754172 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.754173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.755202 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.755204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.755570 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.755572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.755720 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.756163 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.756184 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.757667 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.759295 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.759301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.759303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.759627 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.759808 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.759809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.759810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.759947 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.760166 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.760172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.760320 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.760432 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.760602 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260927 18:38:31.761305 28437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.151.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.151.252@KRBTEST.COM" created.
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:38:31.768440 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.761543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40706)
0927 18:38:31.761581 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.761589 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.761621 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.761858 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.761859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.761988 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.762025 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.762026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.762934 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.762935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.763517 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.763519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.763995 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.763996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.764055 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.764406 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.764409 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.765479 (+ 1070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.767458 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.767459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.767459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.767645 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.767871 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.767872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.767872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.767936 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.768247 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.768248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.768280 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.768328 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.768389 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 18:38:31.768610 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.761599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.5.151.253:37795)
0927 18:38:31.761740 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.761743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.761757 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:38:31.761772 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.761773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.761783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.761858 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.762097 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.762852 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.763613 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.763914 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.763995 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.765567 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.765574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.765574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.765580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.767381 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.767726 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.767730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.767730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.767794 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.768004 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.768005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.768005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.768179 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.768263 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.768368 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.768456 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260927 18:38:31.769079 28437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:31.769429 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.769318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49066)
0927 18:38:31.769363 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.769402 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:38:31.769582 28386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
I20260927 18:38:31.769961 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.769876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49074)
0927 18:38:31.769923 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.769945 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:38:31.770078 28437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:38491: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.151.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab.
Entry for principal kudu/127.5.151.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab.
I20260927 18:38:31.778674 5727 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX68_uN/build/release/bin/kudu
/tmp/dist-test-taskX68_uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:38491
--webserver_interface=127.5.151.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab
--principal=kudu/127.5.151.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.151.212:40513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.151.254:32913,127.5.151.253:37795,127.5.151.252:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:31.819270 28437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } attempt: 1
I20260927 18:38:31.819693 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.819556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49080)
0927 18:38:31.819619 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.819659 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 18:38:31.820052 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.819953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49092)
0927 18:38:31.819998 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.820030 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:38:31.820173 28437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:38491: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
I20260927 18:38:31.823173 28362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } attempt: 3
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
I20260927 18:38:31.829841 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.823467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.5.151.253:37795)
0927 18:38:31.823522 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.823523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.823534 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:38:31.823633 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.823634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.823645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.823713 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.824019 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.824557 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.825236 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.825438 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.825516 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.827257 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.827258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.827259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.827266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.828623 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.829064 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.829065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.829066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.829137 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.829346 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.829347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.829347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.829462 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.829537 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.829685 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.829761 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:38:31.829847 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.823413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40708)
0927 18:38:31.823463 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.823479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.823549 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.823717 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.823718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.823868 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.823929 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.823930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.824634 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.824636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.825146 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.825148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.825519 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.825521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.825598 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.826007 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.826012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.827175 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.828702 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.828703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.828703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.828873 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.829208 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.829209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.829209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.829279 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.829528 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.829529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.829574 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.829661 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.829761 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:38:31.830255 28362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:31.830585 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.830475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49106)
0927 18:38:31.830521 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.830544 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:38:31.830924 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.830848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49110)
0927 18:38:31.830885 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.830906 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 18:38:31.831058 28362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:38491: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
W20260927 18:38:31.853647 28444 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:38:31.853801 28444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:31.853816 28444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:31.853856 28444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:31.854892 28444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:31.854925 28444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:31.854938 28444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:31.854949 28444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:31.854986 28444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:31.856247 28444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.151.212:40513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.151.254:32913,127.5.151.253:37795,127.5.151.252:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/krb5kdc/kudu_127.5.151.252.keytab
--principal=kudu/127.5.151.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.151.252:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.151.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 17:43:27 UTC on ccycloud.quasar-iwwbht.root.comops.site
build id 4676275
I20260927 18:38:31.856485 28444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:31.856696 28444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:31.860822 28450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.860836 28452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.860823 28449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:31.861060 28444 server_base.cc:1034] running on GCE node
I20260927 18:38:31.861315 28444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:31.861608 28444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:31.862175 28362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } attempt: 1
I20260927 18:38:31.862457 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.862361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49112)
0927 18:38:31.862408 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.862432 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:38:31.862736 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.862661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49126)
0927 18:38:31.862697 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.862718 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 18:38:31.862761 28444 hybrid_clock.cc:648] HybridClock initialized: now 1790534311862748 us; error 23 us; skew 500 ppm
W20260927 18:38:31.862821 28362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:38491: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:31.864974 28444 init.cc:377] Logged in from keytab as kudu/127.5.151.252@KRBTEST.COM (short username kudu)
I20260927 18:38:31.865387 28444 webserver.cc:460] Webserver started at http://127.5.151.252:38685/ using document root <none> and password file <none>
I20260927 18:38:31.865619 28444 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:31.865682 28444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:31.865823 28444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:31.866787 28444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data/instance:
uuid: "d51f4566f39048169e3a513d90228364"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "cc366a4e8591568547c9ae086035e7cf"
server_key_iv: "a11773727499a5def09b32dbf8bd39c2"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.867108 28444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/wal/instance:
uuid: "d51f4566f39048169e3a513d90228364"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "cc366a4e8591568547c9ae086035e7cf"
server_key_iv: "a11773727499a5def09b32dbf8bd39c2"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.868273 28444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:38:31.868922 28459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:31.869084 28444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:31.869166 28444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data,/tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/wal
uuid: "d51f4566f39048169e3a513d90228364"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-19nl"
server_key: "cc366a4e8591568547c9ae086035e7cf"
server_key_iv: "a11773727499a5def09b32dbf8bd39c2"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.869241 28444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:31.886417 28444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:31.888300 28437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } attempt: 2
I20260927 18:38:31.888556 28444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:31.888615 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.888506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49128)
0927 18:38:31.888572 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.888596 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 18:38:31.888718 28444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:31.888989 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.888881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49134)
0927 18:38:31.888930 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.888962 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 18:38:31.889098 28437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.151.252:38491: Network error: Client connection negotiation failed: client connection to 127.5.151.252:38491: connect: Connection refused (error 111)
I20260927 18:38:31.892566 28444 rpc_server.cc:307] RPC server started. Bound to: 127.5.151.252:38491
I20260927 18:38:31.892593 28511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.151.252:38491 every 8 connection(s)
I20260927 18:38:31.892899 28444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/data/info.pb
I20260927 18:38:31.893738 28512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:31.895117 28512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:31.902498 5727 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX68_uN/build/release/bin/kudu as pid 28444
I20260927 18:38:31.902618 5727 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX68_uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534055081918-5727-0/minicluster-data/master-2/wal/instance
I20260927 18:38:31.902748 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.895683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.5.151.254:32913)
0927 18:38:31.895732 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.895734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.895925 (+ 191us) server_negotiation.cc:407] Connection header received
0927 18:38:31.896312 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.896313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.896320 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.896370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.896885 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.897316 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.898183 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.898350 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.898397 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.900327 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.900328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.900328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.900334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.901588 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.901884 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.901884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.901885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.901926 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.902129 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.902129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.902130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.902217 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.902262 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.902545 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.902665 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:38:31.902824 5727 external_mini_cluster.cc:1428] Setting key e61c4064afbb7caf6de384224a1fcde5
I20260927 18:38:31.902911 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.895663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.254:32913 (local address 127.0.0.1:41398)
0927 18:38:31.895813 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.895848 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.896242 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.896428 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.896431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.896658 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.896823 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.896826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.897384 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.897384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.898120 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.898121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.898422 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.898423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.898514 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.898999 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.899039 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.900250 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.901651 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.901655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.901657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.901831 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.901975 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.901975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.901975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.902078 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.902266 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.902271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.902413 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.902526 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.902708 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260927 18:38:31.903610 28512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:38:31.909929 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.903828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.253:37795 (local address 127.0.0.1:40720)
0927 18:38:31.903871 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.903880 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.903915 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.904063 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.904064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.904163 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.904208 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.904209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.904836 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.904837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.905450 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.905452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.905867 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.905867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.905909 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.906214 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.906218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.907370 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.908901 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.908902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.908902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.909111 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.909325 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.909326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.909326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.909398 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.909637 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.909638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.909680 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.909756 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.909832 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:38:31.910084 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.903910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.5.151.253:37795)
0927 18:38:31.903953 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.903955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.903962 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:38:31.903983 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.903985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.903995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.904064 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.904276 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.904758 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.905524 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.905802 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.905865 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.907447 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.907449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.907449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.907457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.908816 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.909183 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.909184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.909184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.909255 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.909469 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.909474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.909474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.909574 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.909625 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.909808 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.909881 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:38:31.910439 28512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:38:31.911829 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.903194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.5.151.252:38491)
0927 18:38:31.903315 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.903319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.903453 (+ 134us) server_negotiation.cc:407] Connection header received
0927 18:38:31.903479 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.903481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.903493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.903567 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.904389 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.905197 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.906078 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.906472 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.906559 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.909009 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.909015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.909016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.909022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.910715 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.911037 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.911038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.911039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.911083 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.911299 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.911300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.911300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.911431 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.911592 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.911637 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.911730 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.252@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.254@KRBTEST.COM
I20260927 18:38:31.917806 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.910671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49154)
0927 18:38:31.910723 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.910733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.910779 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.911183 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.911184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.911307 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.911328 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.911329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.912022 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.912023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.912734 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.912736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.913130 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.913131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.913177 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.913534 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.913538 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.914508 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.916732 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.916733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.916733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.916940 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.917149 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.917150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.917150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.917213 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.917478 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.917478 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.917509 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.917564 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.917676 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:38:31.917934 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.910713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.5.151.252:38491)
0927 18:38:31.911067 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.911068 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.911073 (+ 5us) server_negotiation.cc:407] Connection header received
0927 18:38:31.911086 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.911087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.911093 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.911181 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.911402 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.911934 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.912818 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.913051 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.913130 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.914623 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.914624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.914624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.914630 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.916652 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.917015 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.917016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.917017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.917085 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.917272 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.917272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.917273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.917410 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.917484 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.917590 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.917764 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":335,"thread_start_us":70,"threads_started":1}
I20260927 18:38:31.919484 28512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: Bootstrap starting.
I20260927 18:38:31.919531 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.912561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.5.151.254:32913)
0927 18:38:31.912596 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.912598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.912607 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:31.912629 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.912630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.912638 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.912690 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.913116 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.913542 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.914158 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.914383 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.914435 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.916837 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.916838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.916838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.916843 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.918334 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.918764 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.918765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.918765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.918824 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.919010 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.919011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.919011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.919126 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.919315 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.919350 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.919421 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:31.920269 28512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:31.920533 28512 log.cc:824] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:31.921202 28512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: No bootstrap required, opened a new log
I20260927 18:38:31.923211 28512 raft_consensus.cc:348] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
I20260927 18:38:31.923365 28512 raft_consensus.cc:729] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51f4566f39048169e3a513d90228364, State: Initialized, Role: FOLLOWER
I20260927 18:38:31.923458 28512 consensus_queue.cc:260] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
I20260927 18:38:31.923732 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.923779 28512 sys_catalog.cc:564] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:31.923843 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.151.253@KRBTEST.COM
W20260927 18:38:31.925522 28530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:31.925601 28530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:31.927508 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.920138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.5.151.253:37795)
0927 18:38:31.920172 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.920174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.920214 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:38:31.920280 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.920281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.920290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.920342 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.920829 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.921269 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.922144 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.922323 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.922366 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.924823 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.924824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.924824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.924831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.926305 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.926712 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.926713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.926714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.926772 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.926981 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.926981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.926982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.927109 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.927281 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.927350 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.927423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 18:38:31.925976 28444 master_runner.cc:186] Error getting master registration for 127.5.151.254:32913: OK, instance_id { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" instance_seqno: 1790534311598307 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:38:31.927958 28444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:38:31.928674 5727 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:31.929555 28336 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7a5e503f47aa421195a948c89693cf7c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41408
I20260927 18:38:31.937906 28362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } attempt: 2
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.254@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:38:31.944725 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.938159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49164)
0927 18:38:31.938205 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.938217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.938259 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.938391 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.938391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.938528 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.938602 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.938604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.939205 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.939206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.939673 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.939674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.940044 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.940045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.940099 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.940499 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.940503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.941920 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.943692 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.943693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.943693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.943903 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.944094 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.944095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.944096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.944155 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.944440 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.944440 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.944483 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.944551 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.944635 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:38:31.944726 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.938200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.5.151.252:38491)
0927 18:38:31.938256 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.938257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.938266 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:31.938321 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.938322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.938333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.938384 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.938671 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.939128 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.939764 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.939972 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.940045 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.942026 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.942027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.942027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.942034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.943613 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.943976 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.943977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.943978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.944029 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.944215 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.944216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.944216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.944373 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.944436 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.944569 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.944641 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:38:31.945937 28362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c: Bootstrap starting.
I20260927 18:38:31.946586 28362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:31.946906 28362 log.cc:824] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:31.947611 28362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c: No bootstrap required, opened a new log
I20260927 18:38:31.948588 28362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
I20260927 18:38:31.948711 28362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5e503f47aa421195a948c89693cf7c, State: Initialized, Role: FOLLOWER
I20260927 18:38:31.948818 28362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
I20260927 18:38:31.949079 28533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.949131 28362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:31.949154 28533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:31.949257 28336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:38:31.949354 28336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:31.950115 28336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
W20260927 18:38:31.950878 28544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:31.950929 28544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:31.951761 28486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a5e503f47aa421195a948c89693cf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d51f4566f39048169e3a513d90228364"
I20260927 18:38:31.951887 28486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:31.952752 28486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a5e503f47aa421195a948c89693cf7c in term 1.
I20260927 18:38:31.952989 28311 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a5e503f47aa421195a948c89693cf7c, d51f4566f39048169e3a513d90228364; no voters:
I20260927 18:38:31.953214 28336 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [CANDIDATE]: Term 1 election: Requested vote from peers 196e6341474146f685102d52582cf2cf (127.5.151.253:37795), d51f4566f39048169e3a513d90228364 (127.5.151.252:38491)
I20260927 18:38:31.951833 28294 master_runner.cc:186] Error getting master registration for 127.5.151.253:37795: OK, instance_id { permanent_uuid: "196e6341474146f685102d52582cf2cf" instance_seqno: 1790534311748335 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:38:31.954074 28294 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:38:31.954133 28533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:31.954257 28533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [term 1 LEADER]: Becoming Leader. State: Replica: 7a5e503f47aa421195a948c89693cf7c, State: Running, Role: LEADER
I20260927 18:38:31.954332 28533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
I20260927 18:38:31.954738 28533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5e503f47aa421195a948c89693cf7c. Latest consensus state: current_term: 1 leader_uuid: "7a5e503f47aa421195a948c89693cf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.954826 28533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:31.955049 28552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:31.955406 28552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:31.957314 28486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [term 1 FOLLOWER]: Refusing update from remote peer 7a5e503f47aa421195a948c89693cf7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:31.957715 28550 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:31.958992 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5e503f47aa421195a948c89693cf7c. Latest consensus state: current_term: 1 leader_uuid: "7a5e503f47aa421195a948c89693cf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.959092 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:31.966478 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.958792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.5.151.252:38491)
0927 18:38:31.958826 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.958828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.959300 (+ 472us) server_negotiation.cc:407] Connection header received
0927 18:38:31.959400 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.959401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.959411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.959440 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.960176 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.960880 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.961928 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.962239 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.962305 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.963730 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.963731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.963731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.963738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.965336 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.965700 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.965701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.965702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.965757 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.965936 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.965937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.965937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.966080 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.966192 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.966312 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.966391 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 18:38:31.967823 28550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a5e503f47aa421195a948c89693cf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.967906 28550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:31.968309 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a5e503f47aa421195a948c89693cf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.968403 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:31.973850 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.958989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.5.151.254:32913)
0927 18:38:31.959171 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.959173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.966484 (+ 7311us) server_negotiation.cc:407] Connection header received
0927 18:38:31.966681 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.966682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.966693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.966721 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.967307 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.967978 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.968997 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.969284 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.969323 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.970677 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.970678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.970679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.970686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.972336 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.972999 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.973000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.973000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.973091 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.973289 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.973290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.973290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.973405 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.973493 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.973690 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.973762 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260927 18:38:31.977864 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.959418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.5.151.253:37795)
0927 18:38:31.959463 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.959464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.967306 (+ 7842us) server_negotiation.cc:407] Connection header received
0927 18:38:31.967389 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.967390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.967401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.967466 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.968981 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.969677 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.971357 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.972151 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.972196 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.974653 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.974655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.974655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.974662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.976249 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.976833 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.976834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.976834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.976901 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.977190 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.977191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.977191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.977325 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.977399 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.977693 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.977775 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 18:38:31.980221 28437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } attempt: 3
Sep 27 18:38:31 dist-test-slave-19nl krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.151.253@KRBTEST.COM for kudu/127.5.151.252@KRBTEST.COM
I20260927 18:38:31.986312 28552 catalog_manager.cc:1347] Generated new cluster ID: 1b5c73d420c946f0a45e0f9a1611f52c
I20260927 18:38:31.986806 28552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:31.988566 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.980436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.151.252:38491 (local address 127.0.0.1:49194)
0927 18:38:31.980506 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.980515 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.980548 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.980767 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.980768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.980914 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.980967 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.980969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.981899 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.981900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.983184 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.983186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.983645 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.983646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.983697 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.984101 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.984105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.985785 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.987610 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.987611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.987611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.987808 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.987988 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.987988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.987988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.988047 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.988295 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.988296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.988336 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.988392 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.988476 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:38:31.988891 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.980615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.5.151.252:38491)
0927 18:38:31.980650 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.980651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.980658 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:38:31.980677 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.980678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.980689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.980731 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.981063 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.981784 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.983261 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.983545 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.983601 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.985888 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.985889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.985890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.985896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.987510 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.987879 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.987880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.987881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.987927 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.988107 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.988108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.988108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.988222 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.988273 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.988736 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.988805 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:38:31.990327 28437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: Bootstrap starting.
I20260927 18:38:31.991051 28437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:31.991341 28437 log.cc:824] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:31.992161 28437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: No bootstrap required, opened a new log
I20260927 18:38:31.994141 28437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
I20260927 18:38:31.994284 28437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196e6341474146f685102d52582cf2cf, State: Initialized, Role: FOLLOWER
I20260927 18:38:31.994385 28437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } }
I20260927 18:38:31.994764 28437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:31.994855 28410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:31.995695 28410 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [term 1 FOLLOWER]: Refusing update from remote peer 7a5e503f47aa421195a948c89693cf7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:31.995859 28411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a5e503f47aa421195a948c89693cf7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "196e6341474146f685102d52582cf2cf"
W20260927 18:38:31.996202 28390 tablet.cc:2369] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:31.996543 28411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a5e503f47aa421195a948c89693cf7c in term 1.
I20260927 18:38:31.996934 28533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:31.998342 28573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5e503f47aa421195a948c89693cf7c. Latest consensus state: current_term: 1 leader_uuid: "7a5e503f47aa421195a948c89693cf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.998430 28573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:31.998471 28573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a5e503f47aa421195a948c89693cf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:31.998512 28573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:38:31.999246 28572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:38:31.999336 28572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:38:31.996775 28561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:32.000190 28561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:38:32.002974 28574 mvcc.cc:205] Tried to move back new op lower bound from 7334028541775028224 to 7334028541765648384. Current Snapshot: MvccSnapshot[applied={T|T < 7334028541775028224}]
I20260927 18:38:32.016747 28533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a5e503f47aa421195a948c89693cf7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5e503f47aa421195a948c89693cf7c" member_type: VOTER last_known_addr { host: "127.5.151.254" port: 32913 } } peers { permanent_uuid: "196e6341474146f685102d52582cf2cf" member_type: VOTER last_known_addr { host: "127.5.151.253" port: 37795 } } peers { permanent_uuid: "d51f4566f39048169e3a513d90228364" member_type: VOTER last_known_addr { host: "127.5.151.252" port: 38491 } } }
I20260927 18:38:32.016848 28533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:32.018134 28552 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:32.018848 28552 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:32.026484 28552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5e503f47aa421195a948c89693cf7c: Generated new TSK 0
I20260927 18:38:32.026697 28552 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:38:32.047641 28365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.046147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.5.151.254:32913)
0927 18:38:32.046203 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.046206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.047011 (+ 805us) server_negotiation.cc:407] Connection header received
0927 18:38:32.047109 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.047110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.047121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.047167 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.047579 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41428: BlockingRecv error: recv got EOF from 127.0.0.1:41428 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 18:38:32.048336 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.046467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.5.151.253:37795)
0927 18:38:32.046502 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.046504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.047848 (+ 1344us) server_negotiation.cc:407] Connection header received
0927 18:38:32.047963 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.047964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.047973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.048001 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.048276 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40750: BlockingRecv error: recv got EOF from 127.0.0.1:40750 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 18:38:32.049067 28517 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.046781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.5.151.252:38491)
0927 18:38:32.046820 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.046822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.048517 (+ 1695us) server_negotiation.cc:407] Connection header received
0927 18:38:32.048678 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.048679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.048688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.048715 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.049012 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49210: BlockingRecv error: recv got EOF from 127.0.0.1:49210 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 18:38:32.051115 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.050437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.5.151.253:37795)
0927 18:38:32.050465 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.050467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.050688 (+ 221us) server_negotiation.cc:407] Connection header received
0927 18:38:32.050777 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.050778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.050792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.050817 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.051074 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40754: BlockingRecv error: recv got EOF from 127.0.0.1:40754 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 18:38:32.053833 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.053134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.5.151.253:37795)
0927 18:38:32.053163 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.053164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.053384 (+ 220us) server_negotiation.cc:407] Connection header received
0927 18:38:32.053463 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.053463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.053471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.053495 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.053795 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40766: BlockingRecv error: recv got EOF from 127.0.0.1:40766 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 18:38:32.057329 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.056719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.5.151.253:37795)
0927 18:38:32.056746 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.056748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.056933 (+ 185us) server_negotiation.cc:407] Connection header received
0927 18:38:32.057013 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.057014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.057021 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.057045 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.057295 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40780: BlockingRecv error: recv got EOF from 127.0.0.1:40780 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260927 18:38:32.062901 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.062178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.5.151.253:37795)
0927 18:38:32.062205 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.062207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.062399 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:38:32.062478 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.062479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.062486 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.062510 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.062863 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40792: BlockingRecv error: recv got EOF from 127.0.0.1:40792 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 18:38:32.072546 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.071880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.5.151.253:37795)
0927 18:38:32.071908 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.071910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.072116 (+ 206us) server_negotiation.cc:407] Connection header received
0927 18:38:32.072202 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.072203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.072211 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.072237 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.072498 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40808: BlockingRecv error: recv got EOF from 127.0.0.1:40808 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 18:38:32.090338 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.089516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.5.151.253:37795)
0927 18:38:32.089561 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.089562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.089812 (+ 250us) server_negotiation.cc:407] Connection header received
0927 18:38:32.089929 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.089930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.089940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.089969 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.090298 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40824: BlockingRecv error: recv got EOF from 127.0.0.1:40824 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 18:38:32.124441 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.123580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.5.151.253:37795)
0927 18:38:32.123656 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.123659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.123817 (+ 158us) server_negotiation.cc:407] Connection header received
0927 18:38:32.123949 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.123950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.123962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.124034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.124385 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40838: BlockingRecv error: recv got EOF from 127.0.0.1:40838 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 18:38:32.190352 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.189528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.5.151.253:37795)
0927 18:38:32.189595 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.189598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.189734 (+ 136us) server_negotiation.cc:407] Connection header received
0927 18:38:32.189821 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.189822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.189841 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.189922 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.190287 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40840: BlockingRecv error: recv got EOF from 127.0.0.1:40840 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 18:38:32.320075 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.319456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.5.151.253:37795)
0927 18:38:32.319545 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.319547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.319590 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:38:32.319645 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.319645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.319655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.319694 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.320015 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40842: BlockingRecv error: recv got EOF from 127.0.0.1:40842 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 18:38:32.577883 28441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:32.577183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.5.151.253:37795)
0927 18:38:32.577272 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:32.577274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:32.577289 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:38:32.577425 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:32.577426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:32.577445 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:32.577498 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:32.577813 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40858: BlockingRecv error: recv got EOF from 127.0.0.1:40858 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:38:32.925978 28530 catalog_manager.cc:1259] Loaded cluster ID: 1b5c73d420c946f0a45e0f9a1611f52c
I20260927 18:38:32.926040 28530 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: loading cluster ID for follower catalog manager: success
I20260927 18:38:32.926776 28530 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: acquiring CA information for follower catalog manager: success
I20260927 18:38:32.927011 28530 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d51f4566f39048169e3a513d90228364: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:38:32.999760 28572 catalog_manager.cc:1259] Loaded cluster ID: 1b5c73d420c946f0a45e0f9a1611f52c
I20260927 18:38:32.999822 28572 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: loading cluster ID for follower catalog manager: success
I20260927 18:38:33.000568 28572 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: acquiring CA information for follower catalog manager: success
I20260927 18:38:33.000813 28572 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 196e6341474146f685102d52582cf2cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:38:33.093211 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:33.090962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.5.151.253:37795)
0927 18:38:33.091070 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:33.091071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:33.091121 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:38:33.091171 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:33.091172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:33.091231 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:33.091302 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:33.091428 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.091901 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.092580 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.092758 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.092803 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:33.092932 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:33.093042 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:33.093096 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:33.093132 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 18:38:33.093149 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":56,"threads_started":1}
I20260927 18:38:33.097154 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:33.094915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.5.151.252:38491)
0927 18:38:33.095006 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:33.095008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:33.095028 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:38:33.095071 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:33.095071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:33.095104 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:33.095208 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:33.095347 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.095797 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.096481 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.096666 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.096714 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:33.096823 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:33.096922 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:33.096975 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:33.097023 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:38:33.097039 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":46,"threads_started":1}
I20260927 18:38:33.101428 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:33.098837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.5.151.252:38491)
0927 18:38:33.098871 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:33.098872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:33.098881 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:38:33.098962 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:33.098963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:33.098974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:33.099004 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:33.099168 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.099884 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.100747 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.100947 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.101040 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:33.101186 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:33.101213 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:33.101238 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:33.101343 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 18:38:33.101361 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 18:38:33.101833 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:33.098937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.5.151.253:37795)
0927 18:38:33.098974 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:33.098976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:33.099069 (+ 93us) server_negotiation.cc:407] Connection header received
0927 18:38:33.099158 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:33.099159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:33.099167 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:33.099198 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:33.099400 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.100074 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.101100 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.101404 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.101457 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:33.101625 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:33.101654 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:33.101710 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:33.101754 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:38:33.101775 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 18:38:33.102391 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:33.098749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.5.151.254:32913)
0927 18:38:33.098840 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:33.098842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:33.099067 (+ 225us) server_negotiation.cc:407] Connection header received
0927 18:38:33.099114 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:33.099115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:33.099163 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:33.099241 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:33.099368 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.099842 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.101101 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.101332 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.101403 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:33.102071 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:33.102155 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:33.102234 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:33.102290 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:38:33.102313 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":48,"threads_started":1}
I20260927 18:38:33.103976 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 28294
I20260927 18:38:33.108968 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 28369
I20260927 18:38:33.113576 5727 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX68_uN/build/release/bin/kudu with pid 28444
2026-09-27T18:38:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2732 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5423 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (258038 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST